Programming My Life

Django Form Basics

2025-03-07T08:40:06+00:00

Despite using Django for a number of years, I haven’t really worked with Django’s Forms until this week. I needed to create a form to handle a file upload, which has an associated category. The form also had to allow users to create new categories from within the file. When creating a new category, we have some extra data we want associated with the new category. I learned a few things trying to set this up.

First, forms.ModelForm: The fields on this form don't have to map 1-1 to the model. The upload model has a category field, but we need some extra fields on the form when we create new categories that don't exist on the upload model. No problem! I thought I might have to fall back to a standard form since I needed extra fields, but I was able to add them to the form and assign them to the category in the view.

Also, to break up and style different fields in a form, you can individually identify the fields in the template file and style them. So instead of

<formid="id_upload_form" action="{% url 'app_name:upload_file' %}" method="post" enctype="multipart/form-data">
  {% csrf_token %}
  {{ form|crispy }}
  <input type="submit" id="idupload_form_submit" class="btn btn-primary mt-4 disabled" value="Upload">
 </form>

We have:

<formid="id_upload_form" action="{% url 'app_name:upload_file' %}" method="post" enctype="multipart/form-data">
  {% csrf_token %}

  {{ form.file|as_crispy_field }}
  {{ form.category|as_crispy_field }}
  {{ form.new_category_sources|as_crispy_field }}
  {{ form.new_category_display_name|as_crispy_field }}

  <input type="submit" id="idupload_form_submit" class="btn btn-primary mt-4 disabled" value="Upload">
 </form>

This allowed me to wrap the bottom two fields in a Boostrap accordion in order to hide them from users who are updating a category instead of creating one.

<div class="accordion" id="accordionExample">
  <div class="accordion-item">
    <h2 class="accordion-header" id="headingOne">
      <button class="accordion-button collapsed" type="button" data-bs-toggle="collapse" data-bs-target="#collapseOne" aria-expanded="false" aria-controls="collapseOne">
        Create New Category
      </button>
    </h2>
    <div id="collapseOne" class="accordion-collapse collapse" aria-labelledby="headingOne" data-bs-parent="#accordionExample">
      <div class="accordion-body">
        {{ form.new_category_sources|as_crispy_field }}
        {{ form.new_category_display_name|as_crispy_field }}
    </div>
  </div>
</div>

We already have bootstrap as a dependency, so I like this solution better than adding JavaScript, which I also saw as a way to solve this problem.

Pytest, A Practical Guide

2025-02-07T08:40:06+00:00

At a local Python meetup, there were several attendees that weren't familiar with pytest, which gave me the idea to make this post as a primer on how I use pytest. This isn't going to be a comprehensive review of pytest and how to use it. For that, pick up Brian Okken's book.

This post is for someone with knowledge of Python who is maybe just starting to use pytest, or has used it for a while to run tests but hasn't dug into its features. What I want to cover is some of the features of pytest that can help improve your tests and running them. I also want to talk about how to think about pytest because it has some features that are very powerful, but can be confusing if you don't know where to look.

Fixtures

These are a great feature of pytest that can cause some initial confusion.

@pytest.fixture
def first_entry():
    return "a"

def test_first_entry(first_entry):
    assert first_entry == "a"

@pytest.fixture is a marker (more about that below) that identifies the function first_entry as a fixture. It is imported into test_first_entry as a parameter by name.

This is an oversimplified example to illustrate how fixtures work. I've simplified here because what you might find in a real world project is several fixtures being imported into a test where some are defined a few hundred lines above the function, and some might even be defined in another file (see next section). The power of fixtures comes when you need to define variables that will be reused across a number of tests, either in a single file or across the application.

A lot more about fixtures are in the docs.

conftest.py

This is a common filename you will see in projects using pytest. It's a great place to store fixtures and other configuration that will be reused across an entire application (or a section of the app). It's a very helpful pattern, but I've found that it can lead to debugging trouble in larger projects. How so? Two ways:

If you aren't aware that conftest.py exists, you might not be able to track down where something (usually a fixture) is defined.
If you have more than one conftest.py (this is possible and can be a good idea in larger projects), it can also be hard to track down which file things are defined in.

The other reason these are problems is that not all IDEs are able to figure out where fixtures are defined, which means you can't right-click and view their definition, like you can with most objects.

The way I work around this is to use simple project wide string search (ctrl-shift-f or cmd-shift-f in vscode) when I'm looking for fixtures.

Flags

Executing pytest on the command line will run your entire test suite, but I almost never use that command alone. Instead, I use a range of different flags depending on how I am running my tests. Here are the flags I use most often and when I use them.

-n auto

By default, pytest runs on a single core. If you install pytest-xdist alongside pytest and execute pytest -n auto, it runs on all of the cores on your machine. I use this almost every time I run pytest. The only exception is when I'm running a few tests because the overhead isn't worth it in those cases. You can replace auto with a specific number of cores, but I've never had a good reason to do that. Another advantage of running tests this way is catching some flaky tests that might not have cleaned up properly or are not properly isolated.

-x

This stops the test run on the first failure. I use this when I'm running more than one test and I think there is a likelihood that one or more might fail (usually when working through refactoring). Or, if I'm having an issue with my setup, and I think there is a possibility of an error causing many tests to fail. I want the test suite to stop on the first one so I can more easily read the output.

--lf

This runs your last failed tests. I use this when I'm fixing a series of tests and I don't want to wait for the rest of the test suite to run. It can also be helpful if you've broken a handful of tests to work through each one until none fail.

-k name_of_test

This allows you to select a subset of tests. I often use this to run a single test or test file. Note that in larger test suites (thousands, not hundreds of tests) this can take (significant) time to collect. In that case, it's good to use -k to find the test once, then if you are re-running the test a bunch of times, use the full path to the test. You can find the full path by forcing a failure in the test. I do this sometimes when I forget the path syntax which is /path/to/test.py::test_name.

-s

This stops pytest from capturing standard output. It's useful if I need to print something in a test. I try to not do that in favor of using a breakpoint, but sometimes I find this useful.

--pdb

This opens a debugger at the line where a test fails. I should probably use this more often than I do, but I tend to use -x, then introduce a breakpoint(). I did find this useful recently, though. It can be helpful when you're having trouble knowing where a test is going to fail and you know you'll need to poke around at some values when it does.

--reuse-db

This is Django specific (I think) and is great for speeding up your tests in most test runs, but I've run into issues when new migrations are introduced. This is partially specific to my setup because I added this to my pytest.ini, forgot about it, then realized the errors I was getting was because the test database was in a bad state. Once you've done this a few times, you won't forget! This can also happen if you use it by default. If you start getting unexpected testing errors, try running without it. I tend to not use it the first time I run the test suite after pulling down changes, then use it throughout the day that I'm running my tests.

Adam Johnson wrote about his most used flags, which has some similarities and differences with my list.

Related Concepts

coverage.py - this is a great package to use with pytest that tells you how much of your code is covered by tests. This can be very helpful when getting started with a test suite, or when developing new code to ensure you have tested all possibilities. Get started by installing pytest-cov then run with pytest --cov appname. On some projects, I add coverage to my defaults so I don't have to remember to use all of the flags. On larger projects, it can be a bit too much output and I use this Github Action instead.

Markers - we've already seen these with @pytest.fixture above. These are decorators unique to pytest. There are some good built in markers, and you can also define your own, which can be helpful for separating different types of tests (slower vs faster, for example).

@pytest.mark.xfail - A good marker for bugs you know you need to fix. I have also used it for failures we knew would be fixed in future versions of external libraries. That way, we'd see the test pass with the new version, which would show up as a failure in the test run, notifying us to remove the mark, and make any necessary code changes.

@pytest.mark.skip - Another useful marker. This one is a bit like commenting out code, but can be useful for keeping tests around for a short time until they can be updated to no longer require skipping. As such, I use this sparingly.

parametrization - using a marker, you can run a single test with a number of input parameters. This is very helpful for not repeating test code when the only difference is the input. More information in the docs. Unfortunately, there is not a built in way to parametrize fixtures. Fortunately, I have a blog post about how to do it.

With Django

If you are developing a Django application, pytest-django has a lot of nice features for using pytest with Django.

Wrapping Up

I hope this has been helpful in highlighting some useful features of pytest as well as some notable places that you might trip up while using it. Let me know if you learned something new!

Coffee

2025-01-28T08:40:06+00:00

I've been a big fan of coffee for many years, but it's only in the last few that I've really started diving deeper on coffee and its preparation. I don't have an espresso machine, so most of my consumption at home is prepared as a pour over.

I've almost always stuck to two cups per day: one in the morning, and one just after lunch. To make my coffee, I've used a wide range of coffee devices, from a Keurig, to a Chemex, to a Moka Pot, to a standard drip brewing machine (and more). For the past couple of years, though, I've used a Hario V60 for most of my coffee brewing. I made and bought some cold brew in the summer, but most days I use my V60 to make both of my cups. I also have an Orea V3, which highlights slightly different flavors than the V60.

Now, when I get a new bag of beans I tend to try it with the V60 for a cup or two, then the Orea for a cup or two (or vice versa, the order is random). It has been fun to try new coffees in both brewers and see which I prefer. Then I'll change my grind size or temperature a bit to dial it in (if I think it's necessary). This has really given me a better appreciation of coffee!

Coffee flavor preferences

My favorite cups are light roasted, washed coffees with some sweetness and fruit flavor. One of my recent favorites was from grower Jose Salazer roasted by Passenger Coffee which tastes a bit like peach tea. It has a great balance of sweetness and coffee flavor. I've also recently found some naturally processed beans I really enjoyed from Little Waves Coffee.

Similar to changing the brewer above, I've found that experimenting with grind size and water temperature really enhanced my enjoyment of naturally processed coffees. When using too fine a grind and too high temperature (both more suitable for lighter roasted, washed coffees), I was getting too much of the fermented/boozy flavors common to naturals. Using a courser grind and lower temperature water yields great fruit flavors with less of those fermented notes I don't like as much.

I tend to not prefer coffees with floral tasting notes. If a coffee comes highly recommended and has floral tasting notes that are supposed to be subtle, I'll try it. But if a coffee is dominated by floral notes, I tend to avoid them. The exception to this is a good Gesha. I don't get Gesha beans often because they are expensive, but I've had a couple that are very good, interesting and complex.

If you want to chat coffee, please reach out to me!

[TIL] Killing Long Running Postgres Queries

2025-01-17T08:40:06+00:00

I recently had issues with long running postgres operations and found these two commands, one for finding long running queries, and one for killing them. My issue turned out to be with an UPDATE instead of a query, but I'm noting this for future use. Note that the interval can be updated in this query to be more flexible for shorter/longer time frames.

Finding Long Running Queries

SELECT
    pid,
    usename AS username,
    datname AS database_name,
    state,
    now() - pg_stat_activity.query start AS running_time,
    query
FROM
    pg_stat_activity
WHERE
    state = 'active'
    AND now() - pg_stat_activity.query start > interval '5 minutes'
ORDER BY
    running_time DESC;

Killing Long Running Queries

SELECT pg_terminate_backend(<pid>);

[TIL] Shell Alias for Failing Commands

2024-05-03T08:40:06+00:00

TL;DR: Instead of && use ; after a command that might have errors.

My main project at work has a nice Make command for getting a new truncated version of our main database. Unfortunately, I can never remember the exact command, and it requires a second command (pg_restore) to update my local database. It occurred to me that an alias would go a long way to making this easier for me.

alias new_db="make get_new_db && pg_restore other_args"

This was already a big win because now I don't have to wait for the database to download before executing the pg_restore command, which also takes some time. Now I can call the alias and let it run until I have an updated database with no more input.

Recently, I also realized that we have a nice management command that resets all local passwords, so you can set a simpler password locally than for production, so I tried to add that to my alias:

alias new_db="make get_new_db && pg_restore other_args && python manage.py set_pw"

Executing new_db still works, but the final command doesn't run. I failed to find a solution with some basic searching, but a coworker pointed out that the pg_restore command usually has some (expected) errors. The && operator expects the previous command to have an exit code of 0, which this does not. In order to run a command after command with errors, you can use the ; operator so that command becomes:

alias new_db="make get_new_db && pg_restore other_args ; python manage.py set_pw"

And now all of it runs!

TIL: Docker Commands

2024-04-25T08:40:06+00:00

I set up my main project at work to utilize LocalStack this week. I wasn't able to get it working using a virtual environment with Python, but another project at work has it set up through Docker. A coworker that is much more familiar with Docker than I am helped me get set up. Along the way, I saved these Docker commands that have already served me pretty well in debugging/figuring out what's going on in my containers:

docker compose logs -f localstack # see the logs for the localstack container

docker compose up -d localstack # -d flag is for daemon which runs in the background. Can be useful to omit to see if something is failing on startup

docker compose up -d --build web # force a rebuild

docker compose exec web bash # go into shell on the web docker container

docker compose exec localstack awslocal s3 ls s3://localstack-s3 --recursive # view files in the bucket `s3://localstack-s3` in the `localstack` container, recursively.

Mocking in Python tests

2024-03-25T08:40:06+00:00

For a while now, I've felt unsure about mocking in Python. It has helped me test in the past, but I've also seen a lot of smart people talk or write about its pitfalls.

Recently, I had to test some functionality around calls to Twilio, so I decided to investigate to see if I could get a better understanding of the tradeoffs involved.

Blog posts and Conference Talks

This post really clarified things for me. Particularly this part following a section setting up some mocks for testing a couple of API calls (emphasis mine):

And you can imagine adding a few more tests, perhaps one that checks that we do the date-to-isoformat conversion correctly, maybe one that checks we can handle multiple lines. Three tests, one mock each, we’re ok.

The trouble is that it never stays quite that simple does it?

It hit me that the confusion I had with all of these blog posts is that the times I've used mocks, it has stayed that simple (I've also used them in other cases that provide value, but more on that later). This isn't to say that post is wrong. In fact, I agree with the author's point that when things get more complicated, mocks can cause trouble.

Another great example of this is an old (but not outdated) talk from Pycon 2012 in which two engineers from Google talk about how they evolved their codebase to deal with issues caused by large numbers of mocks spread across a large codebase. Ned Batchelder has a good summary of the talk here. If you are having trouble with mocks already, or the number of mocks in your test suite is growing, this is a great talk to see some possible solutions.

Hynek Schlawack has an interesting post about mocking (with a lot of great links for further reading) that covers another pattern I've used: instead of mocking the service you are calling, wrap the call(s) to that service in a function so you can more easily control it in tests. I've found this pattern especially helpful if you are calling a service with different sets of parameters.

I wrote about AWS S3's presigned URL API a few years ago. Back then I only had one way to call the function, as you can see in the example code. Since then, I've added a few files to download, so I'm now calling that API with a few different parameters. I wrapped that API call in a helper function to make testing easier and more robust.

Conclusion for External Web APIs

What I've taken away from all of this is that mocks are like most solutions in technology: they are not a panacea. But they also aren't worthless or always bad. I think Microservices are an appropriate analogy here. They can solve some interesting problems that come with scaling (headcount as well as requests), but their existence doesn't make a monolithic codebase a bad choice. So the next time you are looking to mock object(s) in Python, consider how much you need to mock. Depending on your answer to that, mocks might be helpful, or you might want to look into one of the solutions above.

Use Cases Beyond Web APIs

A lot of the discussion I see around mocks is about mocking external services so you aren't reaching out to a third party service in your tests. But there is another use case I found a lot of value in: skipping long running algorithms to test the code around it. In a previous job, I helped write a library for machine learning functions, a lot of which were machine learning algorithms that took significant time to run (especially compared to the time we'd like for a test to run). Mocking these allowed us to separate our test suite into unit tests that allowed us to test the code around these long running functions (that we ran on CI) and longer running tests that we ran locally and in nightly test runs. This is another good use case for mocks.

Django Tasks - Cronjobs vs Celery

2024-03-01T08:40:06+00:00

Like my previous vs post, this isn't a showdown. Instead, this post is about matching the solution to the problem. The tl;dr is that Celery is great at what it does, but it requires more infrastructure and setup than I needed to solve my problem.

The Problem to Solve

I needed to sync data from my database to Canvas, a learning management system where instructors store grades. If I had a year to do this, I would send the data directly to Canvas as it comes to my server. Unfortunately, I had a much more limited time frame to get this done, and that solution would require a lot more testing and robust error handling in cases where the data does not go through.

Before starting, I knew Celery was a common solution for task based workflows in Django, but I wanted to see what other options there were and how they stack up. I did some research and posted this to see if there was anything I missed in my research, which got some helpful responses.

Based on those responses, I started looking into setting up cron jobs to sync the data. I decided that Celery seems to be a great product that handles a lot of complexity for tasks (long and short running), but it was more effort to set up and maintain than I needed.

Management Commands and Cron Jobs

I created management commands to nail down the code while I looked into libraries for setting up cron jobs with Django. Several were no longer supported, and I didn't find any that I felt confident in. I began looking into how to set the cron jobs up myself in the packer scripts I already had for deployments. Fortunately, it was pretty easy. I created a shell script for each cron job that activates the proper environment on the server (to get the appropriate context to connect to my database and Django application) then runs a management command. These are copied by packer onto the server instance. I referenced those scripts in cronjob.sh which packer executes while setting up the server to set up the cron jobs. It looks like this:

...
(crontab -l 2>/dev/null; echo '11 * * * * /opt/website/cron_canvas_user_sync.sh >> /opt/website/logs/cron_users.log 2>&1') | crontab -
...

Explaining cron syntax is outside the scope of this post, mostly because I don't entirely understand it. Crontab Guru was incredibly helpful in setting these up.

The only (minor) difficulty was having to split my packer script into two so that the cron jobs wouldn't be set up on my staging server.

This has worked great! Initially I had three separate cron jobs running:

One to sync account names on our service with those in Canvas.
One for quiz grades.
One for game progress grades.

The first has since been scoped down a bit, but I still need it for a specific case. I recently combined the last two because they were happening simultaneously and sending data to the same endpoint. I hadn't realized that when I first created them.

In the future, I'd like to set up our server to send the grades directly to Canvas when they come in, but for now, this is working well.

Another alternative to these solutions is Django Q2, which Jeff Triplett has written about. It wasn't as established when I was considering what to adopt, but it's worth considering now.

ramblr

2024-02-23T08:40:06+00:00

In writing for this blog, there are some posts I'd like to write that I don't think quite fit the theme of this blog. For example, I want to write short book reviews to link to from my books page, but I don't want them showing up on the index page and archives. I like having those pages be focused on more technical topics, at least for now.

Introducing ramblr.

One piece of inspiration for this is the blog You Might Find Yourself. I want somewhere I can put comments about links, book reviews, and (some) posts that aren't my usual technical writeups. Currently, I have a good backlog of posts I want to finish. That isn't always the case, though, and I'm hoping these less technical posts keep me in the habit of writing and posting.

I am using Pelican's hidden_articles feature to hide these articles from the index page and archives. I created a new page, ramblr.html to act as a secondary index page for them. This code is a heavily edited version of what was at index.html in my theme.

ramblr.html

{% extends "base.html" %}
{% block content_title %}
{% endblock %}
{% block content %}

{% if hidden_articles %}
{% for article in hidden_articles %}

<section id="content" class="body">
    <ol id="posts-list" class="hfeed">
        <li>
            <article class="hentry">
                <header>
                    <h1 class="entry-title"><a href="{{ SITEURL }}/{{ article.url }}">{{ article.title }}</a>
                    </h1>
                </header>

                <div class="entry-content">
                    {% include 'article_infos.html' %}
                    {{ article.summary }}
                    {% include 'comments.html' %}
                </div><!-- /.entry-content -->
            </article>
        </li>
    </ol>
</section><!-- /#content -->

{% endfor %}

{% endif %}
{% endblock content %}

I also had to update pelicanconf.py with:

DIRECT_TEMPLATES = ['index', 'authors', 'categories', 'tags', 'archives', 'ramblr']

Without this, ramblr.html doesn't generate. If I just add ramblr, all of the other templates don't generate.

Eventually, I might move my ramblr posts to a category or tag, but that requires a lot of fiddly excluding on the index and archives templates and rewriting the categories or tags page to not inherit from the index (or to only inherit for the other categories/tags). I also am not as consistent or structured as I would like to be about categories and tags so I'd like to get those more settled before moving ramblr to one of those.

Keyboards

2024-02-17T08:40:06+00:00

I ~~recently~~* read Timo Zimmermann's post about buying a 'keyboard to last' and it made me realize I haven't posted about my keyboard journey yet. If you don't know much about mechanical keyboards, Brett Cannon's post about deciding on a new keyboard has a lot of great info. It also has some recommendations for ergonomic split keyboards.

I had regular keyboards (the kind that come with a PC or are cheap at Best Buy) for more than a decade of my computing life. At some point, I started hearing more people talk (mostly regarding gaming) about mechanical keyboards and why they were so great. If I remember correctly, a friend let me try his keyboard with Cherry MX Black switches. I liked that well enough so I bought a CM Storm QuickFire Rapid keyboard with Black switches. Rapid is the brand's tenkeyless version, or keyboard without a numpad. Black switches are on the heavier side (require more force to push down) and linear, see Brett's post above for more about these differences. A few years later, I decided I wanted to switch to MX Brown switches and bought the same keyboard with those. They are tactile switches (not linear) and are lighter than the black switches.

I had that keyboard for a few years while I was trying to figure out what I wanted in a keyboard upgrade. I thought I liked the MX Clear switches when they came out, but I used a coworker's keyboard with those switches and found they were heavier than I liked for longer typing sessions. Eventually, I bought a Vortex Pok3r, which is a 60% keyboard (just the letters and modifier keys, no F or arrows keys), but I really missed the arrow keys, so I returned it.

I bought a few keycap sets for my QuickFire while I researched a other layouts, switch types, etc. A few years after I bought the Vortex, another friend of mine was kind enough to let me borrow a couple of keyboards, both with split layouts:

A Nyquist

and an Iris

I started with the Nyquist and decided to really give it a try by using some typing websites to relearn how to touch type using the split orthographic layout (meaning the keys are in a grid rather than offset), which has helped with my overall typing. I tried the Iris a bit, but I preferred the orthographic layout of the Nyquist.

After I returned those keyboards, I started looking for something that fit a few functional and aesthetic criteria. I liked the layout of the Nyquist, but I didn't like the available cases. Eventually I found the Helix. I really like the wood case I was able to get, and the owner of that website provided great customer service.

I also still really liked tactile switches so I went with the (rather expensive) Zeal Zilent 67g switches, which are quite nice. After using those for a couple of years, I wanted to try some linear switches again. I mostly was curious how they would compare to the tactile switches for gaming.

I picked up a set of Zeal Sakurios (also quite expensive) and I ended up switching to them full time. I'd love to test drive a range of other switches, but I don't have a way to do that without spending a lot of money. Maybe over time I'll get around to buying and testing more.

I really love the split keyboard. If I need to use an older standard layout keyboard, I feel like I'm not able to sit as comfortably, and I fumble around with a non-orthographic layout. For some reason, I have fewer issues with my laptop keyboard even though the layout is the same as a standard keyboard.

Another downside to a split orthographic keyboard is the lack of good keycap options to cover every key. I keep seeing sets that don't have 1u (the size of a letter key on a standard keyboard, as opposed to the larger keys like tab, shift, etc.) keys for all of keys I need (such as tab, two shift keys, backspace, raise, lower, etc.).

I've considered getting a second keyboard for gaming, but I do so little gaming with a keyboard and mouse anymore that I think that's unlikely. If I were to buy a gaming keyboard, I'd probably go with the Wooting 60HE for the rapid triggers.

If mine dies or I decide to buy a new keyboard at some point, my current plan is a Nyquist from keeb.io, but I might find something else that interests me more before I get to that point. For now, I love my Helix.

*this one took longer than I expected with the holidays!

[TIL] Using Fixtures with pytest `parametrize`

2024-02-08T08:40:06+00:00

For longer than I care to admit, I was running my end to end tests in only Firefox. I thought it would be trivial to add a fixture to have my tests run using multiple browsers. Unfortunately, there isn't an easy way in pytest to use fixtures as parametrized values*. Here is the solution I landed on:

@pytest.fixture(scope="session")
def firefox():
    return webdriver.Firefox(options=options)

@pytest.fixture(scope="session")
def chrome():
    return webdriver.Chrome()

@pytest.mark.parametrize("browser", ["firefox", "chrome"])
def test_e2e_report(
    browser, request
):
    browser = request.getfixturevalue( # required to use firefox and chrome fixtures
        browser
    )

We are passing string values (["firefox", "chrome"]) into the test with parametrize, then we retrieve the fixture (object) with that name using request.getfixturevalue(browser) built into pytest (link that helped me find this). It's hacky and I don't like it because an IDE won't be able to trace it, but it works!

Related: This doubled the number of tests I was running which started giving me failures because too many connections were open. Running my local server with python manage.py runserver --nothreading stopped those errors from popping up.

*H/T to Jeff Triplett for suggesting pytest-lazy-fixture as an alernative. If I end up doing this in more places in the future, I'd consider using this library. Update: Jeff posted an example of this recently.

Josh Thomas has another way of doing this. Similar to the above, this solution uses strings for the parametrize arguments, but instead of using getfixturevalue, it imports the fixtures as you normally would in a function then uses conditionals (I suppose you could also use a switch statement if you have more parameters) to match the string and the fixture.

Setting up pre-commit to work with WSL and Fork on Windows

2024-02-02T08:40:06+00:00

For the tl;dr instructions, feel free to skip to the last two paragraphs.

Years ago, I used SourceTree for visualizing my git graph. At some point, SourceTree became bloated and I really didn't like the direction it went, so I looked around for alternatives. I forget if I settled on any in between, but for several years now I've been using Fork. I highly recommend it.

For my usage, I like to see what files I've changed and review the changes that are highlighted. I know there are ways to do this in my terminal, but I like this workflow as it gives me a chance to review my changes before pushing in a different window, which helps me take a new perspective.

In a previous job, I got used to using pre-commit with my Python projects to require running black, flake8, and isort. However, I was using a Mac at that job and my personal environment is on Windows, primarily using WSL. I wasn't able to get pre-commit working smoothly between WSL and Windows (where Fork is installed as there isn't a Linux version). At some point, I stopped trying and was running black, flake8, and isort manually. This meant I wasn't running them at all for some commits.

I finally got some time to dive into this and found a way to get everything set up:

Install pre-commit in Windows (not WSL). I did this through git-bash, but you should be able to do it from any terminal (or command line). You may need to install Python first if you haven't already. After installing pre-commit, you will likely have to add the install location to your path (system, not user). If you haven't set your project up to use pre-commit, you may want to manually do that in WSL first as the logs in Fork might make this step annoying if you have to clean up more than one or two issues.

You may need to restart Fork, but now you should be all set!

[TIL] Mocking Chained Calls

2024-01-29T08:40:06+00:00

At work last week, I was asked to write tests for code that reaches out to Twilio's verify service. Here is what that code looks like (from the Twilio docs):

client = Client(account_sid, auth_token)

verification = client.verify \
                     .v2 \
                     .services('VAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX') \
                     .verifications \
                     .create(to='+15017122661', channel='sms')

I was able to use @mock.patch(module.Client)* to mock the client, but I tried a few different ways, unsuccessfully, to patch the return value (or raise an exception) of create() all the way down that chain. After a couple breaks and different incantations entered into Google, I finally stumbled upon what I needed in the docs for unittest.mock, mocking chained calls.

I threw in a bunch of .return_values into my function call, and thought I had solved it, but that unfortunately didn't work. What I realized is that you only need those after the function calls (ones with parenthesis), not the properties (like verifications, v2, etc.). So the final product looks like:

@mock.patch(your_module.Client)
def test_name(mocked_client):
    mocked_client.return_value.verify.v2.services.return_value.verifications.create.return_value = 'return value that I want'
    ...
    # call the function
    # assert some things

If you want to raise an exception:

from twilio.base.exceptions import TwilioRestException
...
@mock.patch(your_module.Client)
def test_name(mocked_client):
    mocked_client.return_value.verify.v2.services.return_value.verifications.create.side_effect = TwilioRestException("", "")
    ...
    # call the function
    # assert some things

*Don't forget to mock the Client in your module, not the library itself in case other tests need the client.

Strategies for a Long Running Side Project

2023-12-19T08:40:06+00:00

At DjangoConUS 2023, I gave a lighting talk with this title. It was a lot of fun, and I'd suggest you try to give a lightning talk if you haven't before. It's a great way to try giving a talk at a conference without having to commit to a full 25 or 45 minute talk.

Here is a link to the talk, if you'd prefer to watch it.

With that intro out of the way, here are my strategies:

Doc

You

Men

Tay

Shun

Documentation!

That's right, I'm saying you should write more documentation for your side project. Why? Context! If you need to stop working on your project, or if you are so deep in the authentication that you haven't touched the business logic in months (for example), you may forget a lot of the context around why your wrote the code the way you did, or what comes next.

But long form docs like these aren't the only kind of documentation you should write. Others include:

Tests

They document how you want your code to work. And how you don't want it to work (failure states you anticipate, bugs that you've found and written a regression test for, etc.). Reading tests can help provide context for the related code. They also mean that if you start writing code that touches a system you forgot about, it could catch an error in your new code. Write more tests! If you'd like to read more about my thoughts on testing, I wrote about that here.

Github Issues

Instead of keeping a plaintext TODO list, I like to use issues to lay out the work, prioritize, and sometimes have discussions with myself about how the work is progressing and what to do next. Simon Willison has a great blog post and talk with more depth about this strategy: The Perfect Commit. In terms of prioritization, I tend to use a Github project to shuffle the issues around which helps me think through longer term planning.

Code comments

I find good comments to be incredibly helpful. Function, method, parameter, variable, etc. names should be self documenting. Comments aren't a substitute for those. But sometimes there is a hack or dense bit of code that needs explaining. Sometimes it's helpful to explain why a particular algorithm was chosen as in the example image. Sometimes this is (or should be) temporary, but sometimes I just can't figure out a way to do the thing that doesn't require at least a little explanation. I like to add comments so the next reader understands that context.

One way I find myself writing comments is that I write some (obviously brilliant) code, then come back to it after a few days | weeks | months and realize it isn't so obviously brilliant. When I remember why I did it that way, I put a comment so future me, or some other (un)lucky developer won't have to wonder.

Side Projects?

But, Andrew, aren’t these great strategies for any project? Yes, but I'm highlighting them for side projects in this post because I think people tend to underestimate how helpful they can be in the long run.

Wrap up

Any strategies I missed, or any thoughts you have about any of these? Let me know! Email the site domain name [at] gmail.com, or reach out to me @programmylife@fosstodon.org

Using SQLite for a Django application on fly.io

2023-11-06T08:40:06+00:00

I'd like to run a few small apps I've been developing for learning purposes (to do list, etc.) on a cheap or free platform as a service (PaaS) provider so I don't have to worry about a full deployment setup like I have on my main project. These projects aren't going to be used by many people (likely just me), and the data is not important. Since managed databases are usually a significant cost, I wanted a cheaper and simpler solution.

After reading about SQLite from fly.io, and seeing some other folks discussing using SQLite for real workloads like this talk from DjangoCon Europe and Simon Willison discussing it at DjangoCon US last year, I thought it would be interesting to use SQLite as a database for a few small apps. With that blog post from fly.io, I assumed it would be easy to get up and running, but it was...not.

I recognize that the fly.io solution for some of this is LiteFS, but for a small app, I'd rather stick with a simpler (and cheaper) solution. I do want to try that eventually and I'll write that up if I find something interesting. Here is a great write up on Litestream and LiteFS on fly if you want to start there.

Setting up the app server

The first issue I ran into is that fly.io defaults to spinning up two app servers, which is the right thing to do for a real production site to avoid downtime. But that means we can't use a single local SQLite file. I think there is a way to share permanent storage between two servers on fly, but the other factor is that I just don't need the redundancy (or complexity) of multiple servers for these apps.

If you already have an app deployed, you can run fly scale count 1 to scale down to a single machine. If you haven't deployed yet, you can use fly launch --ha=false to start with a single app machine. I would prefer if there were a way to add this setting to fly.toml, but I didn't see any way to do that.

Attached Volume Storage

Now that we have a single server, we need to create permanent storage so that we don't lose the database every time we deploy. Here is how you can do that if you have a new app, but remember to add --ha=false to the deploy step or you'll end up with two volumes and have to delete the volumes and app and start over. Since I already had the app set up I followed these instructions. I added the following to fly.toml:

[mounts]
  source="myapp_data"
  destination="/data"

And ran

fly volumes create myapp_data -s 1 -r xxx

where xxx is the region code. and -s 1 is to make the volume 1 GB instead of the default 3 GB. I'm doing this because my application data is going to be very small. You can decide if you'd prefer less or more storage. For some reason if you just add [mounts] to fly.toml and let deploy create the volume for you, it defaults to 1 GB. Also, if you think your app might grow beyond 1 GB, but it will start small, you can always extend your volume with the fly CLI.

Once you create the volume, you'll have to deploy again to get the volume to attach to your server.

Important clarification that I didn't understand immediately from the docs: "myapp_data" is the name of the volume/persistant storage created on fly.io. "/data" is where that volume will be mounted on your machine. That means if you run fly ssh console, you can navigate to your new volume by running cd /data. If you use different names for those settings, remember them for the future commands in this post and refer back here for which is which.

A note on redundancy: Snapshots of permanent volumes are taken once per day (not guaranteed to be at the same time each day) and kept for 5 days. That's good enough for me for these kinds of projects. You can restore a snapshot with the instructions here. If you want a little more backup, you can backup the file locally using fly's sftp command:

flyctl ssh sftp get /data/db.sqlite3 ./db_y_m_d.sqlite3

If you really want to get creative, you could write a cron job to run that command then ship the file to S3/whatever other blob storage. The Litestream docs have some good tips for that.

SQLite

Ok, we have persistant storage that we can access from our server. Now we have to make a few tweaks to use that storage with SQLite properly.

First, we need to tell Django to use our volume storage for the database. For this project I am using environs, so my database setting looks like the following in settings.py:

DATABASES = {
    "default": env.dj_db_url("DATABASE_URL", default="sqlite:///db.sqlite3"),
}

I'd recommend being able to test locally with a SQLite database, so we want to keep the default setting as is. We'll need to update the environment variable in fly by running fly secrets set DATABASE_URL=sqlite:////data/db.sqlite3.

Migrations

Great! One last set of updates. We need to set up the migrate step for our deployment.

If you try to run deploy now, you might get an error django.db.utils.OperationalError: unable to open database file because of this in fly.toml:

[deploy]
  release_command = "python manage.py migrate"

I had this in one project I created but not another. If this isn't in fly.toml, don't worry and create startup.sh as described in the next paragraph.

This runs when the machine does not have access to the persistent volume we created, so we need a way to run migrate later in the process. I removed the line release_command = "python manage.py migrate" in fly.toml and moved the migrate step into my Dockerfile. To do this, I created a shell script at the top level of my project folder (next to Dockerfile, fly.toml, etc.) called startup.sh:

#!/bin/sh
python manage.py migrate
sqlite3 /data/db.sqlite3 'PRAGMA journal_mode=WAL;'
sqlite3 /data/db.sqlite3 'PRAGMA synchronous=1;'
gunicorn --bind :8000 --workers 2 config.wsgi

Note: the last line assumes that wsgi.py is in a folder named 'config'. If you ran django-admin startproject with some other name, substitute that for config.

The sqlite3 lines are from the previously mentioned talk from DjangoCon Europe and require installing sqlite3 and running startup.sh in the Dockerfile:

RUN apt-get update && apt-get install -y \
    sqlite3 \
    && rm -rf /var/lib/apt/lists/*

RUN python manage.py collectstatic --noinput
RUN chmod +x startup.sh
EXPOSE 8000
ENTRYPOINT ["./startup.sh"]

Note: If you have a line in your Dockerfile like CMD ["gunicorn", "--bind", ":8000",...], you'll need to remove it because it is handled in the shell script.

That should be it. Now we can run fly deploy and our app should be using a SQLite file on our permanent volume storage and running migrate each time we deploy!

If you want to make sure everything is set up properly, add some data to your database and run fly deploy again. If you aren't using permanent storage, you're database will be reset. If you are, everything should still be there.

P.S. Thanks to billy for their help on the fly.io community forum for helping me fix a few of the issues I ran into trying to get this all working.

Moving from Django Microframework to Django

2023-10-30T08:40:06+00:00

At DjangoConUS 2019, Carlton Gibson gave a talk called Using Django as a Micro-Framework, where he demonstrated a single file implementation of "Hello, World!" in Django.

At the sprints during DjangoConUS 2023, Paolo Melchiorre showed off an even more minimal version that sparked some great discussion. Specifically, the usage of lambda in this initial version of Paolo's effort made me realize that using the original function based version, you could build from that single file to what an early Django project typically looks like. After a great discussion over lunch with Eric Matthes about this, I decided to write this post to show just that: going from a single file to a project where a user has run startproject and startapp.

I hope this demystifies some of the 'magic' behind Django and we can learn some things in the process. Note that we are using Django 4.2.X for this example.

Getting Started

I forked Will Vincent's repo for django-microframework and have made a few modifications to get us started. First, I renamed hello_django1.py to hello_django.py because we only need one file, and this one was closer to what I wanted to build from. In my fork, the main branch will have the final version of this code, but I've also created branches that start at the beginning and mark important steps in our journey. Here is the first branch.

Writing Blog Posts

We are going to be writing a blog, so let's start with some 'blog posts' in hello_django.py:

from django.conf import settings
from django.core.handlers.wsgi import WSGIHandler
from django.core.management import execute_from_command_line
from django.http import HttpResponse
from django.urls import path

settings.configure(
    ROOT_URLCONF=__name__,
    DEBUG=True,
)

def hello_world(request):
    return HttpResponse("Hello, Django!")

def second_post(request):
    return HttpResponse("This is my second blog post. I'm still figuring this whole thing out.")

def third_post(request):
    return HttpResponse(
        "This is my third blog post. I'm feeling more comfortable with this whole thing, and I'm going to try to write a bit more. "
    )

def fourth_post(request):
    return HttpResponse(
        "Hot dogs are not a sandwich. In this blog post I will first define the meaning of sandwich, then I will ???? which will conclusively prove that hot dogs are indeed not sandwiches."
    )

urlpatterns = [
    path("hello-world", hello_world),
    path("second-post", second_post),
    path("third-post", third_post),
    path("fourth-post", fourth_post),
]

application = WSGIHandler()

if __name__ == "__main__":
    execute_from_command_line()

Here is the branch for that code.

Now that we have those, hello_django.py is getting a bit long, and we need to focus on our writing. What can we do to simplify our code so we can do that? Let's move some of the code from hello_django.py into a few other files.

manage.py

First, we can move the last couple of lines (and the associated import) from hello_django.py to manage.py:

from django.core.management import execute_from_command_line

if __name__ == "__main__":
    execute_from_command_line()

But that yields:

django.core.exceptions.ImproperlyConfigured: Requested setting DEBUG, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

So we define DJANGO_SETTINGS_MODULE and now we have a working manage.py:

from django.core.management import execute_from_command_line
import os

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "hello_django.settings")
    execute_from_command_line()

I'd like to pause here to highlight that a lot of ~~magic~~ misdirection happens when we run python manage.py runserver. In order to understand what is happening, though, we only need to go to the Django source code for execute_from_command_line(). This function handles whatever command you call after python manage.py, such as runserver, makemigrations, and migrate. Walking through these is a post (maybe a series) in and of itself, but at least now we know where to find what happens when we call a command using python manage.py.

Back to our Blog

We still have a few more lines that we can move out of hello_django.py to really focus on our posts. Let's move everything that isn't the posts themselves into their own files starting from the top, moving settings.configure() into settings.py:

from django.conf import settings

settings.configure(
    ROOT_URLCONF="urls",
    DEBUG=True,
)

Now, we can match the change we made to the location of our URLs in settings.py by creating urls.py:

from django.urls import path
from hello_django import hello_world, second_post, third_post, fourth_post

urlpatterns = [
    path("hello-world", hello_world),
    path("second-post", second_post),
    path("third-post", third_post),
    path("fourth-post", fourth_post),
]

Finally, let's create wsgi.py:

from django.core.handlers.wsgi import WSGIHandler

application = WSGIHandler()

Awesome. Now hello_django.py contains only the blog posts.

But we've done something else here, too.

Let's take one final step to see what else we've accomplished. We can make a folder called config (the name isn't important, so you can call this what you want) and move our new files there: wsgi.py, settings.py, and urls.py.

Now we have to match our paths so we update manage.py:

from django.core.management import execute_from_command_line
import os

if __name__ == "__main__":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.settings")
    execute_from_command_line()

and settings.py:

from django.conf import settings

settings.configure(
    ROOT_URLCONF="config.urls",
    DEBUG=True,
)

Here is a link to the third branch with the current state of our code.

We've just done (almost) everything that happens when you create a new django project and execute:

django-admin startproject config .

Two notable exceptions:

First, we have not created asgi.py, which gets created with new projects as of Django 3.0. This isn't an asynchronous project, so we can leave it out.

Second, you may notice our manage.py has less code than the version that Django creates and that we set our DJANGO_SETTINGS_MODULE environment variable in manage.py rather than in wsgi.py. I'm choosing to leave these as is for this exercise to simplify moving from the original code to our destination. I wanted to mention it in case it confuses any readers looking at those files in their own projects, but I don't want to slow us down by diving into every difference between this project and a project where you've run startproject.

After startproject

You may notice that hello_django.py now looks an awful lot like views.py that you're likely familiar with from any other Django project, so let's rename it to reflect that.

Since we're going to want to write longer blog posts than we currently have, it doesn't make much sense to hard code our blog entries. Let's store those in a database. For that, we'll need to set up models.py and add DATABASES to settings.py:

from pathlib import Path
from django.conf import settings

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

settings.configure(
    ROOT_URLCONF="config.urls",
    DEBUG=True,
    DATABASES={
        "default": {
            "ENGINE": "django.db.backends.sqlite3",
            "NAME": BASE_DIR / "db.sqlite3",
        }
    },
)

For the sake of simplicity, here is a minimal models.py:

from django.db import models

class Post(models.Model):
    body = models.TextField()

We've got our model set up, but how can we move our posts into our database? Well, to start with, we can set up our Django admin panel for this project. This requires several changes. First, I added only the required settings here by adding django.contrib.admin to INSTALLED_APPS. then I added each app/middleware entry to address an error message until python manage.py runserver successfully ran without errors.

After those additions, here is settings.py:

from pathlib import Path
from django.conf import settings

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

settings.configure(
    ROOT_URLCONF="config.urls",
    DEBUG=True,
    DATABASES={
        "default": {
            "ENGINE": "django.db.backends.sqlite3",
            "NAME": BASE_DIR / "db.sqlite3",
        }
    },
    INSTALLED_APPS=[
        "django.contrib.admin",
        "django.contrib.auth",
        "django.contrib.contenttypes",
        "django.contrib.messages",
        "django.contrib.sessions",
    ],
    MIDDLEWARE=[
        "django.contrib.sessions.middleware.SessionMiddleware",
        "django.contrib.auth.middleware.AuthenticationMiddleware",
        "django.contrib.messages.middleware.MessageMiddleware",
    ],
    SECRET_KEY="django-insecure-4@r)20xhni#*+xcl*8u0t9b)85djrm+3(^0(7pzbb#7+l337*r",
    TEMPLATES=[
        {
            "BACKEND": "django.template.backends.django.DjangoTemplates",
            "DIRS": [],
            "APP_DIRS": True,
            "OPTIONS": {
                "context_processors": [
                    "django.template.context_processors.debug",
                    "django.template.context_processors.request",
                    "django.contrib.auth.context_processors.auth",
                    "django.contrib.messages.context_processors.messages",
                ],
            },
        },
    ],
)

Installed Apps and Middleware

Before we get to the admin, I want to focus on INSTALLED_APPS and MIDDLEWARE. Similar to running commands with manage.py, these hide a lot of complexity. The upside is that this is where a lot of the 'batteries' are stored that people refer to when saying Django is a batteries included framework. To elaborate, let's start with django.contrib.admin in INSTALLED_APPS. While this is a string in our file, it's actually a reference to code in Django itself. Similarly, you can find the other INSTALLED_APPS by navigating the code there.

Similarly, the strings in the MIDDLEWARE list all refer to classes in the Django codebase that you can navigate to by seeing each element in the string (separated by a period) as a folder in the Django codebase (until the last which is the PascalCase class name). Example: django.contrib.sessions.middleware.SessionMiddleware.

Admin

We are now ready for admin.py:

from django.contrib import admin
from .models import Post

admin.site.register(Post)

startapp

We've hit a bit of a stopping point because we won't be able to see our new model in the admin until we've created and run the migration. In order to do that, we need an installed application for python manage.py makemigrations to pick up the new model. We could run python manage.py startapp to create these files, but we're already most of the way there, so let's create the folder ourselves (call it blog) and move models.py, admin.py, and views.py into it.

We need to update the views import, and add the admin route in urls.py:

from django.contrib import admin
from django.urls import path

from blog.views import hello_world, second_post, third_post, fourth_post

urlpatterns = [
    path("hello-world", hello_world),
    path("second-post", second_post),
    path("third-post", third_post),
    path("fourth-post", fourth_post),
    path("admin/", admin.site.urls),
]

Make and run migrations:

python manage.py makemigrations
python manage.py migrate

We have a functioning admin!

But, in order to log in, we'll have to create a super user:

python manage.py createsuperuser

Ok, now we can move our blog posts into the database. Go to your local admin and log in. Click 'Add' down by 'Posts', and add some text to the field labeled 'Body'. Click 'Save and Add Another' and repeat at least four times.

After we've added all four posts, we can update views.py:

from django.http import HttpResponse
from blog.models import Post

def hello_world(request):
    return HttpResponse(Post.objects.all().get(id=1).body)

def second_post(request):
    return HttpResponse(Post.objects.all().get(id=2).body)

def third_post(request):
    return HttpResponse(Post.objects.all().get(id=3).body)

def fourth_post(request):
    return HttpResponse(Post.objects.all().get(id=4).body)

Run python manage.py runserver and navigate to one of the post URLs (here's our first one), and you can see that we're now getting the text from our database. Hooray!

We're almost done, but since we're good software developers, we should add a test for this project in tests.py:

from django.test import TestCase
from .models import Post

class BlogTests(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.post = Post.objects.create(
            body="post text",
        )

    def test_post_model(self):
        self.assertEqual(self.post.body, "post text")

Run the test.

python manage.py test blog.tests

Finally, for the sake of completion, and to get rid of that warning you've probably been seeing if you're coding along, we'll create blog/apps.py:

from django.apps import AppConfig

class BlogConfig(AppConfig):
    default_auto_field = "django.db.models.BigAutoField"
    name = "blog"

And that's it! Our project now matches a project that has run python manage.py startapp blog.

What have we learned?

When we run django-admin startproject config ., we are creating wsgi.py, settings.py, urls.py, and manage.py in the folder config. Now we know what these do and contain.
If we want to know more about what a manage.py command does, we know we can look into the Django source code.
We can also visit the source code to better understand what any section of INSTALLED_APPS and MIDDLEWARE do.
When we run python manage.py startapp appname, the files apps.py, models.py, admin.py, tests.py and views.py are created in the folder appname.

Aliases, pip, and dotfiles

2023-10-13T08:40:06+00:00

Since my last post on aliases, I've added a few new aliases. Two are from Boost Your Django DX, which I'd highly recommend. The first is a straight cut from the book:

alias pip="python -m pip"

Using python -m pip instead of pip ensures you are using the version of pip for your local python. I've definitely had times where using it instead of just pip fixed issues for me.

The second is:

alias newenv="python3.11 -m venv --prompt . .venv && .venv/bin/pip install -U pip setuptools wheel"

This isn't recommended as an alias in the book, but the pattern is recommended. I set this up as an alias because I've been playing around with a few small projects lately, which has caused me to create and recreate environments repeatedly. This makes it much simpler. The big win for me here is that I really don't like another recommended pattern I see in other places which is python3.11 -m venv venv. The problem with this is that your prompt always looks like:

(venv)computername:~/home

How do you know which environment is active? I rarely have issues related to this since I usually use separate terminals for separate projects, but it provides me peace of mind to know I'm in the correct environment. This brings me to my third new alias:

alias act="source .venv/bin/activate"

Again, I just got tired of typing all of that. And by using --prompt in the previous alias, I can have my prompt match the project while the virtual environment folder name is consistent across all projects. This allows me to run act in any folder where I have a virtual environment and activate it!

In addition to adding these to my development desktop, I also created a dotfiles repo. Now, whenever I set up a new machine (or use my laptop for travel), I can have the same aliases everywhere. Here are two great examples of dotfiles repos: Ned Batchelder and Matt Layman.

Finally, another tip regarding pip. A pip tip, if you will: Set up pip to require an active virtual environment. There are several ways to do this, mentioned in this blog post. If you want to save a click, execute pip config set global.require-virtualenv True. It will set the config file for you automatically. If you want to read more, click through.

The common use case this prevents is accidentally installing a bunch of packages (usually with pip install -r requirements.txt) to your global pip. However, I just ran into another nice use case for this today. I had moved a virtual environment from a subfolder to the main folder of a project (I was experimenting with different ways to create the project so I had a few projects in a single repo). When I tried to use the environment again, pip told me there was no virtual environment active even though I had activated it. I'm not exactly sure what caused the issue, but I wouldn't be surprised if it would have caused more issues down the line. Instead, I just deleted the virtual environment, remade and activated it with my handy new aliases!

Testing Stripe Integration in Django

2023-07-12T08:40:06+00:00

I have been using Stripe to process payments for my company AstroVenture (see more context in a previous post here) for a couple of years now, and it has worked great. One thing that took me quite a while to figure out was how to test that we were properly authenticated with the Stripe API without executing a payment.

I've had two instances that caused issues with purchasing:

One release, I broke the server's local environment so things seemed to be working, but our API keys were not read in correctly. Most things seemed to work, but purchases did not.
Another time I made changes to help secure our servers, but one of the rules prevented most outbound traffic including to Stripe's API!

I wanted a way to test for scenarios like this, but I couldn't find any good solutions until recently. I was searching the API docs for Stripe when I realized I could use any stripe API method that requires authentication to ensure that we are able to access Stripe. If we don't supply the keys or something else goes wrong, the test (below) will fail with an error. I only run this as part of my end to end tests which you can read more about here.

I added an endpoint to my Django application that looks like the following (I removed the authentication I do at this endpoint since it isn't the point of the example, and of course, don't put your API key directly in your code, read it from a secret store or somewhere else secure):

class StripeTest(APIView):
    """
    This endpoint exists so that we can run an end to end test
    for the environment to ensure that we are able to access 
    the Stripe API after we have deployed. 
    """
    def get(self, request):
            stripe.api_key = key
            try:
                data = stripe.Customer.list(limit=3)["data"]
                return Response()
            except Exception as e:
                return Response(str(e), status=status.HTTP_400_BAD_REQUEST)

And in my end to end tests I have the following:

def test_validate_stripe_keys():
    """
    By calling the Stripe API and not receiving an error,
    we validate that the keys are properly loaded and that
    we can access the API. We only care if we get an error,
    so we ignore the return value.
    """
    response = requests.get(
        f"{url}stripe-test-api-is-working/"
    )
    assert response.status_code == status.HTTP_200_OK

Do you have a better way of testing that you can access Stripe after a deployment? Let me know!

Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Testing

2023-06-29T08:40:06+00:00

This is the fourth and final installment of the series breaking down my talk from DjangoConUS 2022. The first entry covered background information about the project, the second was about using Django Rest Framework, and the third was about infrastructure and deployment.

Before diving in, I'd like to emphasize that my testing philosophy isn't the end state for a project. It's a guiding principal for getting started and staying motivated with testing. If you have an established application with a large team, you likely already have rules and processes to follow for testing. What I want to discuss here is how I think about testing as I'm building a project. This isn't limited to small/side projects, but it's much more important earlier in a project's lifecycle.

So what is my philosophy? Test enough to give you confidence to change and deploy your code.

For me, this does not include test driven development (TDD) or a specific coverage number. Why not test driven development? I've tried it, and it doesn't match my preferred way to work in most projects which is to develop my code then write tests. If you find TDD works better for you, definitely do that! I do some development by writing tests first, but I generally write at least some code before I write my tests. And while I do strive for a high coverage percentage, I use that metric more as a guide to see where I might need more tests rather than a bar to meet. That is, I'm more interested in coverage of a given file than I am an overall number.

Ok so why this philosophy? For me, I find that without tests (either at all, or even with limited tests for a specific section of the code), I'm much less confident about changing code. This means that I take a lot longer to write or change code in untested sections of the codebase. Usually, I have to take extra time to think through edge cases since I can't be confident those were covered previously and, I don't know what they are because they aren't written into any tests.

There are several terms specific to testing that I'd like to explicitly define since some folks use different definitions and terminology for different classes of tests. I want to make sure I'm clear about what I mean when I'm using these terms.

First, unit tests test a 'unit' of code, generally meaning as little code as possible (oftentimes a single function) so that you can be sure you are testing each section without depending on other sections, which can introduce complexity into tests.

Integration tests bring together (integrate) two or more 'units' to test their functionality when combined.

End to end tests cover the full functionality of some part of the application. For example, testing a student sending gameplay data to an endpoint and receiving a response, or someone purchasing our game.

Our test suite has a large number of unit tests, almost no integration tests, and a few end to end tests. I'll explain my reasoning for each of these.

First, unit tests. I really like having a good suite of unit tests for two reasons:

Almost every time I write some code, then write unit tests, I find that I refactor the code as I'm writing the tests. Sometimes this is just a little bit of cleanup. Other times, it's a bigger rewrite. But almost every time, writing unit tests gets me to think about my code a little bit differently, and I uncover something that I want to improve.
When I'm making a change to some code I haven't touched in a while, I know that my unit tests will tell me if I broke something. This gives me more confidence to dive in than if I didn't have them.

Unit tests take some time and effort to write and maintain, but I'll take that overhead on any project for the confidence that they give. One other great use of unit tests is covering a bugfix. Sometimes, I'll find a bug, write a fix, then write a test (or two) that covers the case for that bug so we can avoid it in the future.

Why no integration tests?

I think the functionality is covered better by end to end tests for this project. For other projects I've worked on, I've had a much larger suite of integration tests and fewer end to end tests. This is highly dependent on what your application does. Generally, for larger projects, you'll want more integration tests to ensure parts of your code work together without having to run a larger suite of end to end tests for every change.

End to end tests can take a long time to write, a long time to run (relative to unit tests), and are more difficult to maintain. That's why I recommend these only cover the most important parts of your code. Even though these were the most difficult to write and maintain, these give me the most confidence when deploying my code. I know that something can still be wrong if these pass, but I at least know the most important parts of the site are mostly working. I wish I would have written these earlier in the project since my first few deployments were much more stressful without them and required some manual testing.

For our end to end tests, I use Selenium. I've heard a lot of good things about Playwright, and I'm hoping to have some time to look into it, but I haven't investigated it enough yet to recommend it myself. I also use pytest instead of the built in testing system in Django. I don't have a strong opinion about pytest versus the test runner in Django, but I've used pytest a good bit professionally (with and without Django), so I find it easier to get started with. I also like the features it has for running tests (like flags for running just the last failed test, --lf, stopping when hitting a failure --x, etc.) as well as fixtures and parametrized inputs. I'd recommend you give pytest a try if you haven't already. You don't need to worry about the more advanced features when starting with it and you can build up your knowledge as you go.

Speaking of fixtures, I tend to create a factory for each data type that I need (when I need it for a test) with Factory Boy and make that a fixture if I need to use it in multiple tests. I'll move those fixtures out to a conftest file (this is a file particular to pytest) if I find a need for them across multiple test files. That way, I'm not populating conftest with a large number of fixtures that aren't used or are only used in one or two places, making it easier to read. If you'd like me to write more about how I use fixtures, let me know!

Some key advice: write tests early!

This doesn't mean you need 100% coverage on day 1. Or even 10%. But having a test, any test, makes it much easier to write the next one. So start with something as simple as making sure your pages load or if your application is an API, that your endpoints return the correct basic response to a good request. Then, as you are building out your application, keep asking yourself, what sections of the code worry me the most to change, or what do I worry about when we deploy? And write tests in those areas to alleviate your stress. Also, try to get some tests covering your most important code in place as early as you can. In the long run, it'll speed up your development and increase your confidence in deploying your code.

Update for Notarizing Mac Unity Apps

2023-06-26T08:40:06+00:00

I recently received an email from Apple stating that they are migrating users from altool to notarytool to notarize applications. Apple says altool will no longer work for notarization starting November 1 2023, but that applications notarized before that will still behave as normal. I previously wrote about notarizing a Unity app on MacOS, so I wanted to add to that post since the linked scripts will be out of date by the end of the year.

What I'm about to write about is covered by Apple here, but I ended up submitting (and waiting and submitting and waiting) several times before figuring out the proper incantations, so I'm gathering the information here.

First, in the script for notarizing your application, you'll need to change --username and --asc-provider to --apple-id and --team-id respectively. Then you'll need to remove --primary-bundle-id (and the value associated with that flag) and --file. For that last one, remove ONLY --file and leave the location of the file you want to notarize. Finally, swap altool --notarize-app for notarytool submit and add --wait (the wait flag is optional, but I'll explain it below).

So you should go from this in the old script: xcrun altool --notarize-app --username "$USERNAME" --password "$PASSWORD" --asc-provider "$TEAM_ID" --primary-bundle-id "$BUNDLE_ID" --file "$SIGNING_FOLDER/$APP_NAME.zip"

to xcrun notarytool submit --wait --apple-id "$USERNAME" --password "$PASSWORD" --team-id "$TEAM_ID" "$SIGNING_FOLDER/$APP_NAME.zip"

The new tool has a nice progress text in the terminal for the upload status that looks like XX% (YY MB of ZZ MB). The wait flag is nice because it means you can skip the second script (from the link in the previous blog post) for checking the status of the file after you've uploaded it. If you use the wait flag, after the file finishes uploading, you'll see 'In progress ........', while it waits to be notarized. Much nicer than a hanging terminal session with no info!

Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Infrastructure and Deployment

2023-06-22T08:40:06+00:00

This is the third installment of the series breaking down my talk from DjangoConUS 2022. The first entry covered background information about the project and the second was about using Django Rest Framework.

First, some important context: if you are a devops engineer, or have a lot of experience with AWS/GCP/Azure, this post may not be for you. This post is aimed at folks who would prefer to write Django code than deal with the intracacies of deployment.

With that being said, this is the section of the talk that made me want to write this series of posts. Specifically, I realized that I focused a lot on the infrastructure setup of this project, which I want to outline here. However, I wish I would have spent more time focusing on what I think the goal of any successful deployment strategy should be for a Django project regardless of the infrastructure:

Repeatability and confidence in deployments.

There are a lot of ways to get to this point. And on day 1 (or even 100 if it's a side project), you likely won't be there. But starting with good process documentation, and moving that toward making your deployments consistent and repeatable is a massive boost in confidence that increases the time you can spend working on your application rather than its infrastructure. It can also significantly lower your stress when it comes to deploying. Importantly, this is all independent of what services you use to deploy your application!

At AstroVenture, I chose Amazon Web Services (AWS) for our infrastructure early on in founding the company because I had experience with it, and we received free credits for a limited time. At that time, I manually created an EC2 instance for the server (and RDS for the database) and manually installed all of the packages I needed to run the server. I manually installed the app and 'deployments' were done by pulling from GitHub and restarting the gunicorn workers. The 'backup strategy', was the in depth document I wrote with step by step instructions about how I did all of that. I tested it by recreating the server a second time for our production environment and using the first as a test/staging server.

In the event of a catastrophic server issue, I was likely to be down for several hours, if not an entire day. But having that document gave me the confidence that it wouldn't be more than that, and that I wouldn't have to stress that I would miss a step during that recovery. So if you aren't sure how you would bring your servers back in the event of everything going down, I'd highly recommend going through this exercise for whatever service you are using. It can at least alleviate some of the stress of a deployment going wrong.

From there, I hired a friend who had more infrastructure experience to write the Packer and Terraform scripts we use now, and to help me make a few architecture decisions that allow us to have ~zero downtime deployments and scaling. I was already using a load balancer, but we added an autoscaling group so that we can spin up new instances if we need.

The Packer scripts create the server image with the application and all of its dependencies, so if we ever have an issue where we need to redeploy an old stable version, we can do that directly from the image instead of having to recreate it. Luckily, we haven't had to do that yet. We use the Terraform scripts to provision an entirely new server and wait until it is live before swapping it with the previous server (and then terminating that one). There are other tools that handle automating infrastructure and application building that others might prefer, but these have worked well (in combination with some good old fashioned bash scripts) for us.

We also have end to end tests (more on this in a post coming soon), which I run after every deployment to make sure that the most important parts of the site are functioning correctly.

What if you don't have a friend with devops experience that can help you, and you don't have that experience yourself?

There are a number of options of Platform as a Service (PaaS) offerings from companies like Render and Fly.io that a lot of folks in the Django community are using. I'm hoping to try these in the near future along with Django Simple Deploy. So while I can't give specific recommendations for these platforms, I can tell you that the goal: Repeatability and confidence in deployments is even easier to achieve on these platforms than it is on AWS (or GCP or Azure). They handle a lot of the work that our Packer and Terraform scripts do so that you can focus on your application code. The tradeoff with these services is that they can be a little bit (to a lot if you scale very large) more expensive than equivalent 'bare metal' servers from AWS, GCP, or Azure. But they can also be cheaper starting out, and the added price can be worth it while you get started.

No matter what tools you use for hosting and deploying your code, if you are reluctant to deploy because of something in your process you aren't confident about, I strongly recommend you look into ways to address that issue. I found that it was a big relief to stop worrying about deploying once I was able to address the more manual parts of our process.

Finally, remember, you don't have to do this all at once, and you don't have to be at the point of continuous integration of your code to feel confident with your deployments. Take small steps and work toward the goal of feeling confident deploying. It'll make coding a lot more fun!

Lessons Learned Teaching Undergraduate Astronomy with a Video Game - Django vs Django Rest Framework (DRF)

2023-06-15T08:40:06+00:00

If you've ended up here from somewhere outside of this blog, and are looking for an exhaustive comparison of these two libraries, I regret to inform you, this isn't that. If you're here for the next installment of the series breaking down my talk from DjangoConUS 2022, welcome back!

This section of the talk outlines my project's usage of Django Rest Framework (DRF). For some added context, I wanted to start this project using DRF, but ran into some difficulties because the data coming from the game was largely string based and so it wasn't able to be neatly serialized into the types we had in the database. As a result, I ended up using DRF, but I wasn't able to use some of the built in features like viewsets and generics.

If you are new to Django, or have never used DRF, one major reason to consider DRF is if you are interested in returning JSON from your application rather than integrating with the Django frontend. Vanilla Django does a great job with integrating the frontend (templates) and backend code. But if you need to send data to a frontend that's not written in Django, or you are developing an API, DRF has a lot of tools to help.

With that out of the way, there is one bit of this section of the talk that I'd like to amend. In that part, I say that if your data model requires a lot of changes to get from hitting your endpoints to your database (like ours did), that you should consider falling back to vanilla Django (versus powering forward with Django Rest Framework).

What I should have said, is that you can still use DRF, but if you feel like you are fighting with the viewsets or mixins or serializers, you can fall back to the basic APIView, which allows much more prescriptive code rather than the classes built into DRF, which are powerful, but less flexible. If you are more comfortable with vanilla Django, feel free to head back that way, but DRF is capable of handling scenarios where your incoming data don't match your models.

If you are familiar with DRF, you may be thinking 'isn't that the point of the serializer?' and you would be correct. But, handling too much complexity in the serializer can lead to worse performance. Before I was able to rewrite my incoming data, I tried (with some help from a developer more experienced with DRF) to rewrite our endpoints to use DRF and ended up with more queries than I had before! I needed to update the incoming data before I could improve performance. Otherwise, I needed too many queries to gather everything and return it the way it needed to be to match what our game was expecting. I'm hoping to write up more about how I determined the number of queries, and how I keep my codebase from increasing that number. If that's interesting to you, let me know!

This gets us to the core message I want to convey: Don't feel like you have to choose only one way of doing things in a Django project. Or that you can't evolve your codebase in the future.

Yes, consistency can help readability in a project. But just because some of your endpoints use viewsets doesn't mean that every endpoint needs to use viewsets. Use the right tool for the job. It's ok to mix DRF generic views with viewsets, or DRF viewsets with Django class based views if that better matches what you are trying to accomplish for a given route. And you can always evolve your code to better match the libraries you are using as you learn more about them.

One piece of advice from this section of the talk I still find myself coming back to is to really focus on understanding serializers and getting to know the inner workings of DRF (by this I mean what it's doing generally behind the scenes to process requests, not necessarily understanding every line of code in the library). After getting over the initial hump of learning about DRF, it can be easy to write some quick views that work. But it can be harder to modify them if you don't understand some of the basics of what DRF is doing under the hood.

Working with DRF, when I send in data and get an unexpected error, the issue is very often with (what I implemented in) the serializer. Whether that's fields that are missing* (or are not in the model or are misspelled) or I'm trying to do something that isn't possible with a specific serializer class (usually with a serializer that is more specific than what I need). Occasionally, I'll run into an error with a misconfigured URL, or a view that's using the wrong viewset or generic view, but most often, it has to do with the serializer.

*In fact, just today, I hit an error in the admin locally because I forgot to update my serializer after updating a model field's name.

So how do we get to know DRF (and serializers) better? If you don't have an app, build one! Either use the DRF Tutorial or some other tutorial to build an app with DRF. There are many good ones out there (Real Python has quite a few). If you have an app already, try using DRF to rebuild endpoints you already have and see how DRF handles them. Or consider adding a few new endpoints. Think about extra data a user may want, or combining data from multiple tables. Anything different from what the endpoints are currently doing will improve your understanding. Even if these changes aren't optimal, you'll see how changes in the data affect the viewsets/generic views and serializers, which will help your understanding a whole lot more than reading another blog post...

I recently rewrote the majority of the endpoints in my application in DRF, and I love how little code there is in my views.py file now! It's also great that I have fewer queries and less processing to do. A lot of that has to do with the aforementioned update to the incoming data, but it's still great to see.

If you use Django and haven't used DRF (or haven't used it in a while), I highly recommend trying it. There is no need to commit to rewriting your whole application. It can be helpful to simplify endpoints that best match DRF's built in generics or viewsets. And remember that with APIView you can learn as you go and make things a bit more explicit before diving in entirely with generics and viewsets.

AstroVenture and University of Mars

2023-05-17T08:40:06+00:00

In this post, I'd like to provide the context for the project I'm currently writing about on this blog. I'm also using this as an opportunity to start a series of blog posts breaking down my recent talk from DjangoConUS 2022 about what I learned related to Django while building this project.

I've realized since giving the talk there are a few things I'd like to change in/add to it. So instead of a blog post summarizing the talk, I'm opting to revisit the talk and discuss those changes/additions (as well as highlighting the advice I feel holds up). The beginning of the talk covers some of the content of this post. The other sections of the talk will be covered in dedicated posts in the coming weeks.

In 2018, I founded AstroVenture with my former colleagues from Penn State University. When I worked there from 2010-2014, we built a videogame that teaches introductory astronomy to undergraduate students that we later named 'University of Mars'. We founded the company in order to sell that game to anyone who wants to learn astronomy. If you'd like to check it out, the first quarter of the game is free as a demo at the link above. If you'd like to play more than that, contact me! There is also a video at the site if you just want to see some gameplay to get a better idea of how it works.

In the nearly 5 years since we founded the company, this has been a side project that I've worked on actively. It is the project that I really dove into and learned Django with. I'd previously had some professional experience with Django, but I had a lot of other responsibilities on that project, so I didn't have time to dig into Django itself as much as I would have liked. The game was built with Unity3D, which I also learned as I built this project.

The game uses a story to engage students and set up discussions for each topic. The gameplay loop for each lesson is for the game characters to start discussing the topic of the lesson (ranging from gravity to dark energy), then the student explores some type of interactive minigame on the topic, followed by a quiz. Then we usually lead into the next section's discussion and repeat.

The servers that I manage for the company handle the website, which has the functionality you would expect from a Django website: login, download the game, view content related to the game (mostly in the form of our static 'encyclopedia' pages that accompany the game). Additionally, those servers also track student progress as they play through the game in order for instructors to give credit to students for finishing the different sections of the game. Currently, instructors can download a CSV file with player progress and quiz scores, but I'm working on integrating with Canvas and Blackboard to make grade syncing more direct.

The backend code for this player tracking was the first API that I ever wrote and was originally written in PHP hosted in a VPS. When I founded the company, I rewrote the API in Django and moved the hosting to AWS. Since my time was limited, I mostly ported the previous API to Django, faults and all. Fortunately, I had some time to dedicate to this recently and I've just finished updating it to be more efficient and use more of Django Rest Framework's built in features. As for the infrastructure, I chose AWS because I had some experience with it, and I had access to free credits there for a limited time. I'll cover more about the infrastructure in a future post.

We use Sentry for error handling and Stripe for payment processing, and I highly recommend both.

The front end of the website is in a separate repo written in Typescript[1] with a bit of jQuery sprinkled in. It also uses Bootstrap for CSS. I chose to not use Django for the front end because I thought I might have someone else working on it who wouldn't have familiarity with Django. If I were to start again today, I might do it differently and have everything in one repo. I'm not exactly sure what I'd settle on, though I'd investigate HTMX, Alpine.js, and Tailwind for CSS as I've heard good things about those.

If any of these technology choices are interesting to you, I'm happy to discuss them more. See the contact links on the about page, or at the bottom of this page.

In the upcoming posts, I'll talk more about the other sections of the talk: infrastructure, testing, and Django vs DRF.

1 - This is almost worth a blog post in itself. When I chose to use Typescript, it was because I was frustrated with JavaScript's lack of typing leading to errors (usually with JSON objects) and not being able to easily debug them in larger codebases. I think Typescript is good, but it was also difficult to find good resources (responses on StackOverflow etc.) dealing with Typescript outside of React or Angular, which made getting started a lot more difficult for me.

[TIL] Django Admin Permissions, etc.

2023-03-03T08:40:06+00:00

So far in my main project (more on that to come), the only users of the admin have been myself and my cofounder, so I've made us both superusers. Recently, I had a request to allow an instructor to view and update data from their own courses. This would require me to give them admin access, but in a highly limited view. The good news is that there is a lot of information on how to limit admin access in Django! The bad news is that a lot of it applies to entire models or other use cases that were much less specific than mine. Three major things I learned about:

First, I'm not sure if this is the best or only way to do this, but I created a group (I could not get this to work as permissions for only one user) and limited the permissions to just a few models (and only the required permissions). I'd be interested if there is a way to do this per individual, but I think I might need the group settings sooner rather later anyway.

Second, for those models they are able to use, I needed to limit what they could see to only their students. I was able to do that by adding to the relevant models in admin.py:

    def get_queryset(self, request):
        qs = super().get_queryset(request)

        if request.user.is_superuser:
            return qs
        else:
            return qs.filter(institution=their_institution)

This returns all data for superusers, but only returns whatever you filter on to other users. In my case, I only have two levels of admin user, so this simple filter works well. You'll have to fill in the filter depending on your model.

Third, and this was most difficult to track down, I needed to make sure they couldn't see data outside of their scope when adding new students to their course. For the users model, I was able to add a filter to the user admin like above. But for the Course model, I would still see all courses even when using get_queryset for the CourseAdmin as above. I'm not sure why this is. To fix this, I had to use:

    def formfield_for_foreignkey(self, db_field, request, **kwargs):
        if not request.user.is_superuser and db_field.name == "course":
            kwargs["queryset"] = Course.objects.filter(institution=institution)
        return super().formfield_for_foreignkey(db_field, request, **kwargs)

This overwrites the data returned for a foreign key in a dropdown on the 'Add' page, and was exactly what I needed!

Swapping burrs in my Oxo Coffee Grinder

2023-02-28T08:40:06+00:00

And now for something completely different...

You may have noticed my favicon is a mug of coffee. I'm a huge coffee fan, and the last few years I've gone further down the rabbit hole with third wave/specialty coffee.

I recently bought a new coffee grinder to upgrade from my ~5 year old Oxo grinder. I try not to be wasteful, so when considering what to do with the old grinder, I decided to search around to see if anyone had recommendations for modifying it. In my (not so exhaustive) search I found this recommendation for swapping to a better set of burrs which I decided to try.

The instructions were pretty good and the most time consuming part was cleaning out all of the grounds from previous usage (perhaps I should have cleaned it more regularly...). But all in all it was quite an easy swap.

The retention still isn't great, but it's better and the new burrs work well!

Since this isn't a coffee blog (yet...) I don't want to get too deep in the weeds, but the old grinder and the new one have different burr types (flat vs. conical) which has allowed me to highlight the difference between them as I try new coffees. It's a lot of fun!

Signing and Notarizing a Unity App for MacOS

2023-02-24T08:40:06+00:00

Edit: A lot of the information in this post is still great, but Apple have recently recommended moving from altool (referenced in the linked scripts below) to notarytool, so I have added an update to this post here.

For the past few years, Apple's Gatekeeper has made it difficult to run apps downloaded from the internet. Since most of the users of the game that I distribute (more on this soon) aren't technical, we get a significant number of folks needing help even with detailed instructions we supply. I finally had the time to look into code-signing our Unity game for MacOS that would be distributed outside the app store (downloaded from the internet).

The following two links have almost all of the instructions necessary, but I want to highlight two issues I ran into in case others have a similar problem. And I think I can explain the root issue a little better than I found elsewhere.

Links: a good example of how to script the process and a walkthrough with much more thorough explanations

Note that my Macbook is managed by the IT department at my current employer, so this may be an issue only for managed machines.

First, it isn't specified in either link, but I installed my certificate (the one I created and downloaded from developer.apple.com, see the second link above for more about that) into my 'login' keychain after running into the issue below and reading through some apple developer forum and stack overflow discussions. I'm not sure if that matters, but it seems that is the recommended way.

I kept getting Warning: unable to build chain to self-signed root for signer when trying to run codesign, and the answer here about the WWDR Intermediate Certification worked for me initially. I installed the linked cert into my System keychain. But I was using the wrong cerification (Distribution, which is for submitting to the store, I think) so notarization failed.

After getting the correct certification (again, from the instructions in the second link above, and installing into my 'login' keychain), I had to download the intermediate certification Developer ID - G2 (Expiring 09/17/2031 00:00:00 UTC) from here, and I installed that into my System keychain. This got notarization to work!

I know it was this one because after I added that, my certification (the one I created in my developer account, downloaded and installed to login) changed to 'trusted'. I tried a different one first that didn't change the status of my certification.

In short, if codesigning is giving you the error above, you are probably missing the intermediate certification from Apple (this was another answer from the developer forum link above, but I didn't understand it when I read it). What this means is that you should determine which type of certification you requested from Apple (Apple Distribution, etc.) and find the matching certification from Apple. It wasn't immediately clear to me which was the matching certification in my case, but I downloaded two that seemed like they might be correct and only one of them changed my cert to trusted in Keychain Access. Good luck!

Using Docker for Debugging

2023-01-31T08:40:06+00:00

My current set of tools for deploying my application to production includes Packer and Terraform. I didn't write all of the code for the deployments, but I've been over most of it now.

When trying to upgrade my server from Ubuntu 20.01 to 22.01, I ran into some problems with conflicting versions of dependencies (related to postgres, mostly). My first instinct was to create an EC2 instance, then walk through each step manually, but I realized I didn't even have to spin up an instance if I could use Docker.

I'm not much of a Docker user, but I've used it a few times professionally. Mostly other people have done the hard work of creating a docker image, and I've run it for development. So I thought this was a great opportunity to try using it myself.

I started by spinning up a docker image for the target Ubuntu version, named jammy:

docker pull ubuntu:jammy
docker create -it --name jelly ubuntu:jammy
docker start jelly
docker attach jelly

Then running through the scripts from my packer build manually:

echo "deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main"> /etc/apt/sources.list.d/pgdg.list
apt-get update
apt-get install -y wget gnupg
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -

export DEBIAN_FRONTEND=noninteractive
apt update && apt upgrade -y && apt auto-remove -y && apt update

apt-get install -y unzip libpq-dev postgresql-client-14 nginx curl awscli logrotate

This is where I get the errors about conflicting versions or versions not available. Did you catch it?

echo "deb http://apt.postgresql.org/pub/repos/apt/ bionic-pgdg main"> /etc/apt/sources.list.d/pgdg.list

The bionic in there refers to the version of Ubuntu that we are requesting the dependencies for. It should be jammy!

This was a great example where Docker saved me some time (and a few pennies) from not having to spin up a cloud instance. And there really wasn't much to learn about Docker itself to dive in. So my advice to you is to give Docker a try for a simple use case like this. Despite being used in massive (sometimes very complicated) build chains, Docker is a relatively simple technology to get started with if you already have some familiarity with the Linux version you are planning to use in your container.

Maybe someday I'll start using it for building a small project...

Bash Aliases

2023-01-24T08:40:06+00:00

This is a short post to tell you to create more bash aliases.

I have had one for my devops setup for a while because I have to set several environment variables to get just about anything done. More recently, though, I set up a few more for my main Django project and even this blog. For the blog it's a simple as entering the directory and activating the environment:

alias pmlstart='cd ~/pml_blog && source env/bin/activate'

For the Django project, it took me forever to remember how to start my local postgres instance (is it service start, or start service, or...?), so I finally just made that part of an alias for when I start working on the project:

alias webstart="cd ~/website && source env/bin/activate && sudo service postgresql start && source env_vars.sh"

Since writing this, I added that last piece to it to load environment variables into my environment so I don't have to remember to edit my activate file when I need to recreate my environment locally. Highly recommended!

Are there bash commands you use frequently together? Make an alias!

Code Formatting Configs in Django

2023-01-21T08:40:06+00:00

There are a few code formatting tools I like to use in just about any Python (and Django) project: black, isort, and flake8. These all serve slightly different purposes, and there are alternatives to each. A full discussion/comparison of code formatting tools in Python is beyond the scope of this post*, but in brief:

Black is great because it automatically formats my code with a well recognized style. I have my editor (currently VSCode) set to run black when I save a file. This makes my code consistent, and is great in teams I've worked in because it means we don't have to worry about styling in code reviews.
isort is great for sorting imports. Even though I tend to do a bit of this sorting as I go, I inevitably swap an import or leave one out of place. This organizes them so I don't have to.
flake8 catches minor (and sometimes not so minor) code issues like unused variables.

There are a few ways to configure these libraries, but for now, I have them in their own individual config files. These should all be placed at the root of your Django project.

For black, I use the default config. In some other projects, I've extended the line length, but currently I'm leaving black as default.

For isort, isort.cfg is the filename and mine looks like this:

[settings]
skip_glob=env/*
extend_skip_glob=**/migrations
profile=black

Where env is the name of your environment. The extended skip for migrations isn't necessary, but I wanted to avoid changing my migration files. The black profile is nice to avoid conflicts between black and isort. Skipping your environment is likely required. When I've left this out (or accidentally run isort without the config) isort runs against the libaries in my environment and ends up breaking them by creating circular dependencies. One way to make sure you aren't going to do this is to run isort --check-only . to make sure it's only running against your files before actually running it.

For flake8 .flake8 is the filename and mine looks like this:

[flake8]
exclude =
    migrations
    __pycache__
    manage.py
    settings.py
    env
    .pytest_cache
    .vscode

These exclude all of the files in the folders migrations, __pycache__ and env (where env is the name of your environment). You may want to exclude more, or not exclude some of these, but I've found this to work for me.

Finally, on other projects I've used all of these with pre-commit, which is a library that uses git hooks to prevent you from committing code that doesn't use the standards you and your team have set, such as black, isort, and flake8, or some other combination of code formatting tools. It also allows you to use a single command to run all three tools.

Unfortunately, I can't figure out how to use pre-commit in my current setup with my python projects in WSL and my git client (Fork) in Windows. Perhaps this is a sign I should start using the git CLI more. But if you know of a good way to use precommit with this setup, let me know!

*between writing and publishing this I read James Bennett covering similar ground. He goes a bit deeper on some of the tools I mention (and other related tools), so I wanted to link this for further reading.

Switching from Jekyll to Pelican

2022-11-07T08:40:06+00:00

In the time since I've posted on this blog, I've erased the environment I used to set up and run Jekyll. Since I've never been a Ruby programmer and most of the code I write now is in Python, I thought I'd look into a blogging engine in Python. I found Pelican and Nikola to be recommended by a number of folks, so I decided to give Pelican a try, and it's been great so far! Using a Python based site generator feels more comfortable for me since I'm more familiar with pip than gems.

It seems remarkably similar to Jekyll and I only had to make a few small changes to the front matter on a few posts to get all of my content working with Pelican. I've made some tweaks to the default theme to move a few icons around to my liking, but otherwise haven't made major changes. I'll make a follow up post if I make interesting changes that others might want to use.

Accessing WebPacked TypeScript from Javascript

2019-06-15T08:40:06+00:00

While I was trying to incorporate Stripe’s Javascript library into my codebase, I ran into some issues implementing it in Typescript. I eventually was able to fix those issues, but while I was trying to figure that out, I left the Stripe code as Javascript and stumbled upon an interesting way to interface that with my Typescript code compiled by WebPack.

First, at the bottom of the Typescript file you want to call from Javascript:

module.exports = {
    methodName: methodName,
}

In webpack.common.js:

  output: {
    ...
    library: 'MyExposedNamespace'
  }

Finally, in the Javascript file:

MyExposedNamespace.methodName();

AWS presigned URLS in Django

2019-04-13T08:40:06+00:00

I found the documentation for presigned URLS on AWS using boto to be insufficient, so here is how I formed the request.

I wanted to have a file private in S3, but expose it to a user for download if they were authorized to access it. I found the easiest way to do that was leave the object in S3 as private, but use the AWS API to generate a pre-signed URL with a low timeout (a minute or so). Unfortunately, I found the documentation not so straightforward and cobbled together a few Stack Overflow answers to come up with exactly what I needed. In particular, I needed to add the ‘ResponseContentType’ to get the file to download correctly, and needed to specify my AWS credentials.

I’m using Django to serve up the download via a get request:

class GetDownloadURL(APIView):

    def get(self, request):
        # Get the service client.
        session = boto3.session.Session(profile_name="AWSUserName")
        s3 = session.client("s3")

        # Generate the URL to get 'key-name' from 'bucket-name'
        url = s3.generate_presigned_url(
            ClientMethod="get_object",
            Params={
                "Bucket": "your-s3-bucket",
                "Key": "SampleDLZip.zip",
                "ResponseContentType": "application/zip",
            },
            ExpiresIn=100,
        )

        return Response(url)

Notes: You will need to update the Key and Bucket params to match what you have in S3. Depending how you have set up your AWS credentials, you may be able to omit the ‘profile_name="AWSUserName"’ parameter. I prefer to be explicit in my config because I’ve run into issues when I have used the default in the past.

Jekyll Setup for PML

2019-03-31T08:40:06+00:00

I originally created this blog using an EC2 instance on AWS and used that to run Jekyll and host the site. Recently, I shut down that instance in order to move the site generation to my local machine(s) and host the site on S3. This article was very helpful to me when setting up HTTPS with S3

For now, my Jekyll setup is pretty basic, and I’ve only made a handful of changes:

-I created a pages folder to hold the few non-blog-post pages (currently just About, Books, and Projects) to clean up the home directory. I accidentally moved the homepage into the pages folder at one point and was very annoyed and confused at my homepage not updating when I’d rebuild with changes. Consider yourself warned!

-Instead of building using the command line, I created a simple bash script to build the site then send the changes to S3:

#!/bin/bash

#set destination and source so that we can run from anywhere
#helpful when editing posts in the _posts directory
JEKYLL_ENV=production bundle exec jekyll build -d /mnt/c/blog_location/_site -s /mnt/c/blog_location

#remove this script from the site directory.
rm /mnt/c/blog_location/build-blog.sh

#upload to s3
aws s3 sync --delete --size-only /mnt/c/blog_location//_site/ s3://programmingmylife.com/ --profile MyProfile

The file paths are /mnt/c/ because I typically build the site from my local Windows Subsystem for Linux install.

-Added google analytics. I tried following a blog post to do this, but it broke the page (I must have mistyped something on the includes), but it made me realize jekyll now has GA built in! You just need to add the line: ‘google-analytics: UA-11111111-1 ‘ to your _config.yml (where UA-11111111-1 is your GA tracking ID) and set JEKYLL_ENV=production before building.

-I got rid of categories in the URLs, which is the default. This was as easy as adding ‘permalink: /:year/:month/:day/:title.html’ to _config.yml

-I also wanted to add the full contents of posts to my home page. I couldn’t find a config setting to change this, so I had to edit the minima theme. First, I had to find the theme files with

bundle show minima

Then, I edited _layouts/home.html to include all post content as shown here. I may change this as the number of posts increases.

I’d like to improve the styling on the blog or get a different theme at some point, but for now, it is mostly the current (at the time of posting) default for Jekyll.

Setting up Webpack to Modify URL per Environment in TypeScript

2018-10-17T08:40:06+00:00

A website I am currently creating has a very simple front end, but I wanted to be able to swap out instances of my API URLs in TypeScript depending which environment I am working in or building for. So far they are all ‘http://127.0.0.1:8000/’, but when I deploy, I didn’t want to have to remember to set a URL for prod or staging etc.

Unfortunately, I was unable to find a good way to do this with just the TypeScript compiler for static files. I’m not a big fan of the complexity that webpack adds, but I’ve noticed it also minifies and obfuscates my javascript, which is a nice bonus. If you know of a good way to do this with the TypeScript compiler alone, please let me know!

I tried a few different methods, but a combination of using webpack and splitting my config into development and production and DefinePlugin worked. Otherwise, I mostly just followed the installation instructions for webpack including installing locally. I tried installing globally based on other tutorials, but I ran into a bunch of issues doing that.

Also, I’m a little frustrated with how confusing the documentation was on this. I tried using Environment Variables in Webpack, but I couldn’t figure out how to access those in my code (not just the config file). It also took me some searching to understand how to use DefinePlugin because the documentation does not make it clear where that should go or how to include it. I found these two links that helped me figure it out (see my webpack.dev.js, webpack.staging.js, and webpack.prod.js files below).

With this setup, I run ‘npm run build:test’, ‘npm run build:staging’, or ‘npm run build:prod’ depending if I am working locally or building for production. Those commands are mapped in package.json:

{
  "name": "av_frontend",
  "version": "1.0.0",
  "description": "",
  "private": true,
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build:test": "webpack --config webpack.dev.js --watch ",
    "build:staging": "webpack --config webpack.staging.js",
    "build:prod": "webpack --config webpack.prod.js"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@types/bootstrap": "^4.1.2",
    "@types/node": "^10.12.12",
    "bootstrap": "^4.1.3"
  },
  "devDependencies": {
    "ts-loader": "^5.3.1",
    "typescript": "^3.2.2",
    "webpack": "^4.27.1",
    "webpack-cli": "^3.1.2",
    "webpack-merge": "^4.1.4"
  }
}

I added --watch on test so that when I’m developing in VSCode, I can just leave it running and it’ll update whenever I save a file. If I want to run that manually I have to run ‘node_modules/.bin/webpack --config webpack.dev.js --watch’ because I installed webpack locally, not globally.

When I first got this set up, I realized it was only outputting a single Javascript file because I didn’t understand how ‘entry’ and ‘output’ worked in the webpack config file (see code below). Now I define each entry JS file for each page (they both have an import for my config with the IP addresses) with a name. Under ‘output’ [name].js corresponds with each ‘entry’. So my output ends up in the dist/ folder and the two files are named ‘login.js’ and ‘purchase.js’ based on the two fields in the ‘entry’ object. Any files added to 'entry' will produce a corresponding output Javascript file.

I also missed something in the instructions for Typescript when I was initially setting this up that lead to a long hunt for why it wasn’t including my config.ts file (the error messages were not great). Don’t forget the ‘resolve’ field below or webpack will get confused when trying to import any file with a .ts extension referenced in another file.

webpack.common.js

const path = require('path');

module.exports = {
  entry: {    
    login: './src/login.ts',
    purchase: './src/purchase.ts'   
  },
  devtool: 'inline-source-map',
  module: {
    rules: [
      {
        test: /\.tsx?$/,
        use: 'ts-loader',
        exclude: /node_modules/
      }
    ]
  },
  resolve: {
    extensions: [ '.tsx', '.ts', '.js' ]
  },
  output: {
    filename: '[name].js',
    path: path.resolve(__dirname, 'dist')
  }
};

webpack.dev.js

const merge = require('webpack-merge');
const webpack = require('webpack');
const common = require('./webpack.common.js');

module.exports = merge(common, {
  mode: 'development',
  plugins: [
    new webpack.DefinePlugin({
      'process.env': {
        'API_URL': JSON.stringify("http://127.0.0.1:8000/")
      }
    })
  ]
});

webpack.staging.js

const merge = require('webpack-merge');
const webpack = require('webpack');
const common = require('./webpack.common.js');

module.exports = merge(common, {
  mode: 'production',
  plugins: [
    new webpack.DefinePlugin({
      'process.env': {
        'API_URL': JSON.stringify("https://staging.com/")
      }
    })
  ]
});

webpack.prod.js

const merge = require('webpack-merge');
const webpack = require('webpack');
const common = require('./webpack.common.js');

module.exports = merge(common, {
  mode: 'production',
  plugins: [
    new webpack.DefinePlugin({
      'process.env': {
        'API_URL': JSON.stringify("https://production.com")
      }
    })
  ]
});

I wrote this blog post originally thinking I had solved this problem, but the solution I was using can only handle development and production environments. It also is a little more complicated than the above solution. It is described here: https://basarat.gitbooks.io/typescript/content/docs/tips/build-toggles.html

Debugging PostgreSQL Port 5433 and Column Does Not Exist Error

2018-10-16T08:40:06+00:00

I am creating a Django application using PostgreSQL (PSQL) for my database and was nearly finished with the API when I discovered some strange behavior. After successfully testing the API in the Django app, I decided to run some basic queries on the database. I received the following error for nearly every field in the app:

    select MaxSceneKey from game_progress_gameplaykeys;
    ERROR:  column "maxscenekey" does not exist
    LINE 1: select MaxSceneKey from game_progress_gameplaykeys;
                   ^
    HINT:  Perhaps you meant to reference the column "game_progress_gameplaykeys.MaxSceneKey".

I was getting the same result for every field in the table that I tried (and when I try to include the table name as the hint suggests), except for ‘user_id’ and ‘objective'.

I confirmed that the fields existed using \d+ game_progress_gameplaykeys, tried changing some of their field types, and even upgraded from Postgres 9.5 to 10.5 (I was planning to do this anyway).

After a bunch of searching, I found the issue:

“All identifiers (including column names) that are not double-quoted are folded to lower case in PostgreSQL.” from https://stackoverflow.com/questions/20878932/are-postgresql-column-names-case-sensitive

I created camelCase field names in my Django app based on what the field names were previously in my application (written in C#).

I decided to fix this (for now) by fixing my models to all use snake_case and using https://github.com/vbabiy/djangorestframework-camel-case to switch the keys from camelCase to snake_case when they come into the API. One issue solved!

While debugging that issue, I decided to update my laptop’s code + postgres version since I hadn’t worked on it in a while and wanted to see if the issue was just on my desktop. When I reinstalled PSQL, I couldn’t seem to log into it using the user I was creating. Using the postgres user was fine, though.

I finally figured out the issue was that PSQL was running on port 5433, not 5432 (the default). After that, I was puzzling over what could be running on 5432 since ‘netstat’ and ‘lsof’ revealed nothing else running on my WSL Ubuntu VM. As I was searching around, I saw someone mention that really only PSQL should be running on that port, and I realized I had installed PSQL on Windows on that machine before I moved over to WSL. I uninstalled that, switched back to 5432 in Linux, restarted PSQL, and boom, good to go.

While I was debugging that issue, I learned some good information about PSQL along the way:

/etc/postgresql/10/main/postgresql.conf allows you to set and check the port that PSQL is running on.

/etc/postgresql/10/main/pg_hba.conf allows you to set different security protocols for connections to PSQL. Notable for local development: set the local connection lines to ‘trust’ so you don’t have to enter a password when logging in.

Note: you need to restart the PSQL server for either of these changes to take effect. Note 2: MORE IMPORTANT NOTE: Don’t use trust anywhere other than a local version of PSQL. Ever.

These are the lines I had to change to get that to work (may be different in versions of PSQL other than 10.5):

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

Configuring NGINX for localhost

2018-10-14T12:50:06+00:00

I had a little trouble finding a simple way to set NGINX up to work locally, so I wanted to write up some quick instructions here. I’m using NGINX with Windows Subsystem for Linux (WSL).

First, I installed NGINX in WSL with ‘sudo apt-get install nginx’

Then, I created a symlink to my frontend directory in my home directory in WSL.

In /etc/nginx/conf.d, I created basic config file localhost.conf:

server {
    listen       8080;

    location / {
        root   /home/username/frontend_directory;
        index  index.html index.htm;
    }

    # redirect server error pages to the static page /50x.html
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
}

The only thing you should need to change is the bolded frontend_directory. Keep in mind the path may differ depending where you keep your files. I restructured my front end after setting up Webpack to include a dist/ folder and broke this config before modifying this line again.

To start NGINX (I have to do this every time I restart the computer): ‘sudo nginx’

Then go to 127.0.0.1:8080 and your site should be live!

If you need to make changes to your configuration, you should first try: ‘sudo service nginx reload’ which will give you a ‘hot’ reload instead of restarting the server.

If you do need to restart the server, you can do so with ‘sudo service nginx restart’.

Sending JSON to a server using fetch() in TypeScript

2018-10-11T12:50:06+00:00

For a new project, I wanted to use TypeScript on the front end but not any of the frameworks that usually include it (React, Angular, etc.). Unfortunately, this means that when I have been trying to figure out how to do something in TypeScript, searches often lead me to solutions involving those frameworks.

I still haven’t found a good resource for creating a JSON object and sending it to a backend using TypeScript. The easiest solution would be to relax the TypeScript compiler and writing it the same way we would in JavaScript, but that defeats the point of using TypeScript. In looking at example code, I found that creating an interface to describe the JSON object is one accepted way to do it.

interface IJSON
{
    email:string;
    fullName: string; 
    shortName: string; 
    password: string; 
    institution: string; 
    isStudent: boolean;
}

const url = 'http://127.0.0.1:8000/register/';

function gatherData(e:Event)
{
    e.preventDefault();  //don't reload page so that we can test.

    let json:IJSON = 
    {
        email: (<HTMLInputElement>document.getElementById("email")).value,
        fullName: (<HTMLInputElement>document.getElementById("fullName")).value,
        shortName: (<HTMLInputElement>document.getElementById("shortName")).value,
        password: (<HTMLInputElement>document.getElementById("password")).value,
        institution: (<HTMLInputElement>document.getElementById("institution")).value,
        isStudent: true,
    }
    sendDataViaFetch(json);
}

function sendDataViaFetch(json:IJSON)
{    
    var request = new Request(url, {
        method: 'POST',
        body: JSON.stringify(json),
        headers: new Headers({
            'Content-Type': 'application/json',
            'Authorization': this.basic })
    });

    fetch(request)
    .then(function() {
        // Handle response we get from the API
    });
}

window.addEventListener('submit', gatherData);

If you have a better way, please let me know!

Python Reference Talks

2018-10-10T12:50:06+00:00

While trying to get more familiar with Django, I started watching talks from DjangoCon from the last few years. I can’t seem to find the talk, but one of them had a list of great Python/Django talks, which inspired me to create me own list (with some definite overlap).

I have found that revisiting talks like these make me reconsider some design problems that I have recently worked through, so I want to keep a list and rewatch these periodically. I will likely add to this list in the future.

These two go together (from DjangoCon2015): - Jane Austen on Pep 8 - Lacey Williams Henschel - The Other Hard Problem: Lessons and Advice on Naming Things by Jacob Burch

Setting up Conda, Django, and PostgreSQL in Windows Subsystem for Linux

2018-08-19T12:50:06+00:00

Because I feel much more comfortable in a terminal than on the Windows command line (or in Powershell), I’ve really been enjoying Windows Subsystem for Linux (WSL). In fact, I use it almost exclusively for accessing the server I run this blog from. WSL is essentially a Linux VM of only a terminal shell in Windows (with no GUI access to Linux) and no lag (like you get in most VMs).

When I created my Grocery List Flask App, I began by using WSL. However, I ran into an issue that prevented me from seeing a locally hosted version of the API in Windows, so I switched to the Windows command line for that app.

Recently, I’ve been developing a Django application (more on that in a future post), and I ran into a similar issue. Between posting the issue with localhost on WSL and starting this new app, there was a response that I had been meaning to check out. I found that for Django and PostgreSQL, making sure everything was running from localhost (or 0.0.0.0) instead of 127.0.0.x, seemed to fix any issues I had. PSQL gave me some issues just running within WSL, but I found that I just need to add '-h localhost' to get it to run.

Below are the commands I used to get Conda, Django, and PSQL all set up on my PC and then again my laptop. This works for Django 2.0, PSQL 9.5, and Conda 4.5.9.

Installation Instructions

Edit: I originally had installation instructions in here for PSQL 9.5. If you want 9.5 in Ubuntu, good news! You already have it. To install the newest version of PSQL, you should uninstall that version first, then install the new version from here

Install Conda (need to restart after installing for it to recognize ‘conda’ commands)

#create environment
conda create --name NameOfEnvironment
#activate environment
source/conda activate NameOfEnvironment
#install Django
conda install -c anaconda django
#install psycopg2, to interface with PSQL
conda install -c anaconda psycopg2

If you get ‘permission denied’, or it hangs, just rerun the install command that failed. Not sure why, but that fixed things for me.

#Remove PSQL from Ubuntu:
sudo apt-get --purge remove postgresql\*
#Then run this to make sure you didn’t miss any:
dpkg -l | grep postgres

#Install PSQL 10 using instructions here: https://www.postgresql.org/download/linux/ubuntu/ I have copied them here for convenience, but please double check that they have not changed
#Create the file /etc/apt/sources.list.d/pgdg.list and add the following line:
#deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main

#Then exeucte the following three commands
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install postgresql-10

sudo service postgresql start    
sudo -i -u postgres -h localhost
createuser --interactive
       psql user: local_user
       y to superuser

#create the database
createdb local_db
#log into local_db
psql -d local_db

#privileges for Django to modify tables.
GRANT ALL PRIVILEGES ON DATABASE local_db TO local_user;

ALTER USER local_user WITH PASSWORD 'password';

'\q' to quit interactive console.
'exit' to leave postgres as postgres user.

#one line command to log in as the user to check tables during development.
psql -h localhost -d local_db -U local_user

python manage.py makemigrations
python manage.py migrate

Now log back in to PSQL using the line above, then enter '\dt' and you should see tables like django_admin_log, django_content_type, django_migrations, and django_sessions. Your PSQL DB is now connected to your Django app!

#optional for now, but allows you to ensure db connection works by storing credentials for the superuser you create.
python manage.py createsuperuser

#command to run the server. go to localhost:8000 in your web browser to view!
python manage.py runserver 0.0.0.0:8000

I used this post for reference

Unity3D Scriptable Objects

2018-07-15T12:50:06+00:00

This week at our local Unity user meetup group, I presented (along with a co-organizer of the group) about Scriptable Objects in Unity. You can find that talk here.

This is that same content in text form.

Scriptable objects are a powerful tool in designing and developing games in Unity3D. It took me longer than I’d like to admit to get around to using them, but I’d like to introduce them in such a way that makes it easier for you to just get started using them.

What is a Scriptable Object (SO)?

It is a script that derives from ScriptableObject, instead of MonoBehaviour. This script allows the user to create objects either in memory or as .asset files in Unity, which are also referred to as Scriptable Objects.. A simple example:

using UnityEngine;

[CreateAssetMenu(menuName = "SOs/FloatVar")]
public class FloatVariable : ScriptableObject
{
    public float Value;

    void MethodName()
    {
        //Do stuff
    }
}

The line with ‘CreateAssetMenu’ adds a new line to the ‘Create’ menu in the Project window in Unity. When you click that menu item, it will create a new .asset file that has access to the variables and methods defined in your file.

It does not have access to the standard Update(), Start(), Awake()* methods because those are part of MonoBehaviour. It does derive from Unity’s Object class, so it has access to classes like GameObject, Transform, etc.

*use OnEnable for initialization instead of Start or Awake

SOs can contain data and functions just like a MB, but it can’t be attached to a GameObject in the hierarchy as a component. A SO can be referenced by a MB, though.

Two things to differentiate:

A script that creates the SO (same as a MB in Project) The SO, which is a .asset file. (lives in Project folder, analogous to an instance of a MB in the hierarchy).

SOs aren’t meant to replace MBs everywhere in your project. But there are places where they are a better fit for storing data/functions.

Why use SOs?

No need for JSON, XML, Text, which means no need to parse data. Can save large amounts of data and optimize data loading when you need it. They don’t get reset when exiting playmode! Since SOs aren’t tied to the scene (not in the hierarchy), you can commit changes to source control without impacting a scene another team member may be working on. This allows you to more easily reference data/functions across scenes when using multiple scenes in development, which I highly recommend you do (this could be a whole other blog post). No need to depend on manager classes to hold all of your references.

3 and 4 combine to allow us to store the data and functions for a type of enemy and tweak that inside of play mode, have the changes saved, then share that with a teammate without having to worry about impacting the scene file. We also don’t have to re-instantiate prefabs or change every instance of the enemy in a scene (or multiple scenes).

You may already be doing something similar with prefabs (holding/referencing data and never instantiating that particular prefab). If so, look at SOs! Using prefabs for this purpose is confusing and accident prone (accidentally throw a prefab into a scene, get confused between what is a prefab and what is a data holder).

If you are a less experienced Unity developer and this seems like a lot to consider, don’t worry about digesting all of it. Just think about some piece of your game design and try to make it as a SO instead of a Monobehaviour such as an enemies stats, or some inventory items.

If you are more experienced, but some of this doesn’t entirely make sense, please try out SOs for some small use cases to see how they differ from MBs. It took me some time to get used to thinking in terms of using SOs, but they are a great tool for a lot of use cases.

How to use SOs

A few Unity Learn examples that demonstrate different use cases for SOs:
Text Adventure
Ability System
Character Select
Customizing UI

Two talks about SOs that really helped me understand how and where to use them:
Richard Fine - Overthrowing the MonoBehaviour Tyranny in a Glorious Scriptable Object Revolution
Link to the project from the talk

Game Architecture with Scriptable Objects
Blog post for the previous talk

Serialization

This is how Unity reads out your data attached in scripts. This gets talked about alongside Scriptable Objects sometimes because the serialization that Unity does sometimes messes up scriptable objects. Unity serializes data when it enters/exits play mode and some data types don’t play nice (polymorphic classes for example). If you are having issues with data resetting/corrupting under those circumstances, check these out:
Forum post on Serialization and Scriptable Objects
Blog post from Lucas Meijer
Blog Post from Tim Cooper
Talk by Richard Fine

Grocery List App and Flask Deployment Issues

2018-07-01T12:50:06+00:00

In addition to starting this blog, I wanted to build some small projects to get some experience with technologies I am not currently using at work. Since I’m currently using Django at work, I decided to create a small Grocery List application using Flask and DynamoDB. You can find the repo and installation info here.

I ran into two major issues when trying to create the application:

First, I tried to set things up without a virtual environment for Python, which caused errors with libraries not pointing to the correct locations. I thought that since this was the only application I’d have on the server, a virtual environment wouldn’t be as important. I realize now there is a big upside to separating your application Python install from your system Python install. I highly recommend setting up a virtual environment for your Python app whether it be virtualenv, conda, or something else even if you only intend it to be the only app on a system.

Second, I didn’t understand how to set up virtual hosts with Apache when I started this project. Getting this blog, the front page of the grocery list app, and the Flask API all routed correctly and running simultaneously took me a few hours to figure out. Two things that seemed to be required to get this all running (otherwise, I was getting the top root folder serving up on all subdomains):

'NameVirtualHost *:80' at the top of the file and the port number (80) in this line for each of the host definitions: ''. See link below for that line in context.

As I was working through the issues, I created this thread for help on reddit: https://www.reddit.com/r/flask/comments/7fbfs1/af_apache_deployment_questions/

I almost posted to serverfault.com, but I ended up figuring things out as I was creating a post.

References: http://peatiscoding.me/geek-stuff/mod_wsgi-apache-virtualenv/ https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xvii-deployment-on-linux-even-on-the-raspberry-pi

Backups Using a Network Attached Raspberry Pi

2018-03-10T12:50:06+00:00

After setting up my Raspberry Pi as a NAS, I wanted to set up backups that are easy to run and check on. Initially, I wanted to set them up to run automatically, but another goal of the Pi set up was for me to turn my PC off more often. I’m currently thinking that if I’m going to turn the PC off, I will just run the backups manually since turning my PC on every Saturday night (or whenever I’d set it to run) isn’t really automated. If I go back on this and set up a cron job I’ll be sure to post about that as well.

One problem I haven’t been able to solve yet is how to backup Windows itself with this set up. Windows 7 backup tool fails, and I can’t see my network drives with the two free backup applications I tried for Windows. I can include specific folders from that drive in my backup script and/or occasionally switch my external drive to my PC to run that full backup, which is probably what I’ll end up doing.

Apart from Windows, I have a few different backups I want to run: 1. The SD card for my raspberry pi 2. A large hard drive with a lot of media files (movies, music, pictures, etc.) 3. A SSD that has all of my games on it. 4. Also on that SSD are folders for my personal software and game development projects. I want to also back these up to AWS.

Here is the script as it currently stands. I run it from Windows Subsystem for Linux on Windows 10. I have some notes below to explain the set up and why I chose to use the tools and configurations that I did.

#!/bin/bash
today=`date '+%Y_%m_%d'`;

#backup raspberry pi
ssh username@ipaddress "sudo dd if=/dev/mmcblk0 bs=1M | gzip - | dd of=/media/pi/HDDName/pibackup/pibackup$today.gz" > /mnt/e/rsynclogs/pibackuplog$today.txt

#backing up all of my development work including Unity to S3 for offsite backups. Have to add dates to the log files otherwise it overwrites the file
aws s3 sync /mnt/e/Development s3://developmentFolder/ --delete > /mnt/e/rsynclogs/S3DevOutput$today.txt

aws s3 sync /mnt/e/Unity s3://unityFolder/ --delete > /mnt/e/rsynclogs/S3UnityOutput$today.txt

#backup D drive excluding a few folders, and write logs out.
rsync -avP --delete --size-only --exclude-from '/mnt/d/rsynclogs/exclude.txt' --log-file=/mnt/d/rsynclogs/rsynclog$today.txt /mnt/d/ username@ipaddress:/media/pi/MediaBackup/

#backup E drive excluding a few folders, and write logs out.
rsync -avW --delete --size-only --exclude-from '/mnt/e/rsynclogs/exclude.txt' --log-file=/mnt/e/rsynclogs/rsynclog$today.txt /mnt/e/ username@ipaddress:/media/pi/GamesBackup/

The Raspberry Pi backup is modified from: https://johnatilano.com/2016/11/25/use-ssh-and-dd-to-remotely-backup-a-raspberry-pi/

I don't have access to the network drives from the terminal (or at least I don't know how to access them from WSL without ssh ing), so I needed the output to be relative to the Pi. The quotations enclose the commands that get sent to the Pi, so I had to extend them to include the output location. I also changed 'bs=1m' to 'bs=1M'. I believe the lowercase m is expected on Mac, but the uppercase is required on most flavors of Linux.

In order to run it from the script I had to set up my user to not require a password to execute the command, which I did by doing the following:

At a terminal on the Pi, enter 'sudo visudo', change the last line to: 'username ALL = NOPASSWD: ALL', where username is the username you are using to ssh. If you are doing this as the pi user, I don’t think this will be necessary. I'd kind of like to limit this to just the ‘dd’ command, but I'm not sure how to tell it where dd. I may update this in the future.

For setting up rsync with the correct flags, I used these two links: https://www.howtogeek.com/175008/the-non-beginners-guide-to-syncing-data-with-rsync/ https://www.thegeekstuff.com/2011/01/rsync-exclude-files-and-folders/?utm_source=feedburner

Two notes for the rsync commands:

By default, drives mount with the 'pi' user. Since I was setting up my backups to work with a different user, rsync was giving me errors about not being able to set the time on the files when I'd run the command. I’m pretty sure this was because the user didn’t have permissions on the drive. By adding the drives to fstab, it mounts them as root instead, which allows the user to access them since it has root permissions. I should have done this when setting up the drive as a NAS, but I only did it for the initial drive I was testing. See here for instructions on adding drives to fstab: https://www.howtogeek.com/139433/how-to-turn-a-raspberry-pi-into-a-low-power-network-storage-device/

For my E drive rsync, I tried initially with the same settings as the D drive, but the backup kept hanging on different files. I saw several recommendations of different flags that people claimed to be the culprit. I tried turning several off and on, but the one that seemed to fix things was swapping -P for -W (suggested here: https://github.com/Microsoft/WSL/issues/2138), which forces entire files to be transferred instead of partial files. I could probably add --progress back in, but -v for verbose gives me enough output to see where issues arise. I’d advise adding --progress back in if you encounter issues and need to check where things are going wrong..

You can find instructions for setting up the AWS CLI tools and using syncing with S3 in the AWS docs. I couldn’t find anything in there for logging, but StackOverflow had a good solution: https://stackoverflow.com/questions/35075668/output-aws-cli-sync-results-to-a-txt-file

The last thing I added to the script was a variable to grab the current date so I don’t overwrite the pi backup or the log files.

One thing I’d like to add is a way to clean up the pi backups. At ~3GB each, it isn’t a big issue currently, but eventually I’ll want to clean them up.

Setting up a Raspberry Pi as a NAS and Plex server

2018-03-01T12:50:06+00:00

When my external HDD failed, I debated getting a network attached storage device (NAS) before realizing the price wasn’t worth it for me. I don’t have that much data, and really all I wanted was a way to automate backups and have a plex server that requires less power than my PC (so I could turn it off more often).

While I was looking around at options, I found I could do both of those things with a Raspberry Pi. I’d been wanting to get one for a while, but never had a good project to justify picking one up. I ordered a Raspberry Pi 3, a case with a fan and a power supply (that has a power switch), and a 32GB SD card. That is more storage than I need, but a 16GB card wasn’t much cheaper. I also picked up an 8 TB Seagate external hard drive.

While I think anyone can set this up, I will say that I have a decent working knowledge of Linux, which helped getting started and troubleshooting issues I ran into. If you aren’t very familiar with Linux and the terminal, you can still get all of this set up, but debugging issues and working through it all might be a little more difficult.

First, I would suggest setting up SSH on your pi so you don’t have to go back and forth between working on the pi and another machine: https://www.raspberrypi.org/documentation/remote-access/ssh/ I’d also recommend setting it up with an ssh key so you don’t have to enter your PW every time you log in: https://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/

I didn’t do either of those at first and it became annoying switching back and forth between machines since I only have one keyboard/mouse/display set up. If you have a separate keyboard, mouse, and display for your Pi, it might not be as helpful right away, but I’d still recommend it.

My first step was to try to set up the NAS. I ran through this tutorial: https://www.howtogeek.com/139433/how-to-turn-a-raspberry-pi-into-a-low-power-network-storage-device/, but I hit a wall near the end. I think this was written for Windows 7 and I’m on Windows 10 where mapping a network drive looks different. I also think I may have skipped the last step of setting the samba PW for my user account, which may have been the bigger problem. Here are some notes about where I did things a little differently:

My external HDD had 4 partitions when I started the process and they automatically mounted, so I skipped the parts about mkdir /media/USBHDD1 and mount...USBHDD1

'security = user' was not already in the samba config file commented out so I just added it in the authentication section. For the section they tell you to add to the config file, I made four copies of that at the bottom of the file, one for each partition that I have.

For the last part of the tutorial about adding the network drives on my Windows PC I had to follow different directions and go to Windows Explorer -> This PC -> Map network drive (in the file menu) -> then in the file for folder enter '\raspberrypi\', then I clicked 'Browse' which let me select the folder (I divided my external HDD into a few partitions). I was also to manually enter '\raspberrypi\nameOfFolder' to get it to see a drive. I repeated that for each of my drive partitions.

One mistake I made was only setting up one of the partitions in fstab. This caused me some serious issues with permissions when trying to set up backups with rsync.

I’m in the process of automating my backups. When I finish testing my backup scripts, I’ll be sure to post about it.

To set my pi up as a plex server, I followed this tutorial: https://www.codedonut.com/raspberry-pi/raspberry-pi-plex-media-server/

The only problem I ran into was that I had to change permissions on the folder ‘/media/pi’ where it automounted my drives because Plex couldn't access them. The permissions on the drive folders themselves were fine.

After that small adjustment, I was able to stream a 1080p movie with a bitrate ~10Mb/s over my local network without any trouble, but I tried streaming one closer to 25 Mb/s and the Pi definitely couldn’t handle it. I’m not sure where exactly the limit is, but that is something to note.

Finally, I wanted to test how resilient this set up is so I made sure I could restart while ssh’d and even shut down the pi.

To restart from command line: ‘sudo reboot’. This allowed me log back in after a couple of minutes.

To shutdown then start up the pi: ‘sudo halt’ on the command line. This shuts it down, but the red light stays on (and the fan, so the board is still getting power). I can then use the power button on the AC cable to shut off power, then press it again to turn it back on. When it comes back up, the NAS drives automount and I can see them on my PC and Plex is running.

One last thing I did, for security, was to create a user other than pi, give it sudo permissions (‘sudo usermod -a -G sudo USERNAME’), and give the ‘pi’ user a much more complicated password so it would be more difficult to hack. I saw one tutorial recommend deleting the pi user account, but I decided that was overkill. At the very least, you should change the default password of you default user if you are going to make the Pi visible on a network.

Unity 2D Tools for Level Building

2018-02-24T12:50:06+00:00

This week for our local Unity meetup group, I presented an intro to some of the new 2D Tools in Unity (There is an intro about more general Unity topics, so for the 2D stuff skip to 17 minutes in): https://www.youtube.com/watch?v=xopzxmzFJUs

Here are the links to things I mentioned were outside of the scope of that talk but might be interesting to learn:

Sprite masks: https://docs.unity3d.com/Manual/class-SpriteMask.html

2D side scrolling brawler style camera (focus on 9-slicing sprites and new features for sorting): https://unity3d.com/learn/tutorials/topics/2d-game-creation/introduction-and-goals

Platformer character controller: https://unity3d.com/learn/tutorials/topics/2d-game-creation/intro-and-session-goals

https://github.com/MelvynMay/UnityPhysics2D - a lot of interesting scenes demoing 2D physics.

Pretty cool topdown game from Unity to show off tilemap and other 2d features (from a talk at Unite Austin, >https://www.youtube.com/watch?v=RkaEh--qUAY>): https://github.com/Unity-Technologies/2d-gamedemo-robodash

2D Game Kit - This is a 2D Game that Unity built to show off 2D features, and what a complete project looks like including tools for designers so that they don’t need to dive into the code to create new puzzles, levels, etc. https://blogs.unity3d.com/2018/02/13/introducing-2d-game-kit-learn-unity-with-drag-and-drop/, https://unity3d.com/learn/tutorials/s/2d-game-kit. Unity also recorded a live training for this recently that I’m assuming they will publish soon, but I can’t find a link to it yet.

Edit: Unity posted the live training for 2D Game Kit here: https://unity3d.com/learn/tutorials/projects/2d-game-kit/overview-and-goals?playlist=49633

What I covered in the video is using the new Tilemaps and associated features for designing levels in 2D. This was also covered by this Unity Learn tutorial: https://unity3d.com/learn/tutorials/topics/2d-game-creation/intro-2d-world-building-w-tilemap, and this blog post: https://blogs.unity3d.com/2018/01/25/2d-tilemap-asset-workflow-from-image-to-level/. This are very thorough and a great reference for these features. I found that there were a couple of things I could talk about not covered in those videos, specifically how to create your own rule and random tiles, and how to create tiles and tilemaps from art that you generate or find yourself.

Finally, here is the collection of brushes and tiles that Unity has coded that cover a huge range of use cases: https://github.com/Unity-Technologies/2d-extras

The ground sprites I used came from here: https://bitcan.itch.io/tileset-simples

And the flower sprites I used came from here: https://onimaru.itch.io/green-platform