Crash Course - Web Dev with Flask, Heroku, SQLAlchemy, Bootstrap

Now we've got our "Flask on Heroku" working let's take a brief intermission to look at what you've created.

Take a look look at your Heroku dashboard https://dashboard.heroku.com/apps

Try these Heroku commands to help you get an idea how Heroku dynos work and practice using the built-in help:

heroku ps
heroku help ps
    
heroku logs
heroku help logs

heroku run bash
heroku help run

One last thing I would like you to do is to understand the @app.route('/') code we used. Here it is again:

@app.route('/')
def index():
    return '<h1>Hello World!</h1>'

This is a Python feature called a decorator, it means that when index() is called it will actually be passed to app.route.

Let's look at the python help for app.route. Assuming you still have your virtualenv activated, run python3 and then run these commands inside to get the help for that:

from flask import Flask
app = Flask(__name__)
help(app.route)

That should print the help for app.route so you can find out how this decorator works.

Our project's current directory structure has the Python code at the top level mixed up with all the other random files. This is a mess.

In order to tidy this up we will make a Python package.

mkdir hello
git mv hello.py hello/__init__.py

Now commit that and push it upto Heroku. You will see that it works the same as before.

Over-time as your code grows and you divide bits out you will evolve a healthy tree of modules and packages within your hello directory. See this article from the Flask docs (n.b. instead of "hello" they called their app "flaskr"): Project Layout

In addition to local testing you will want a staging environment. A test environment in Heroku to validate your app works as expected before pushing changes to production.

This is managed by adding extra git remotes, linked to different sites.

First rename the remote we made before to have a less generic name. I use "prod" for "production".

git remote rename heroku prod

Create the new site. It's a staging env so I call it $APPNAME-stage, with a git remote called "stage".

heroku create $APPNAME-stage -r stage
git remote -v

Cool so we now have two remotes, one called "prod" and one called "stage". Remember these as you will use these names to push all future changes like so:

git push stage main:main
git push prod main:main

Eventually you may want to add these two apps to a pipeline. I don't want to bloat this article too much so I invite you to add this to your TODO-list of things to read later: Pipelines | Heroku Dev Center.

Alright. Heroku exposes "Config Vars" to your application, which Python can access easily using os.environ['VARIABLE_NAME']. I call them "environment variables".

Here is why you need environment variables:

• things like Passwords and API keys should never be committed to your git repo, so they will be in variables
• your staging environment will have different URLs for it's database and other Heroku Add-ons
• your app can know which environment it is so it can do things such as enabling debugging, or displaying a banner to remind you it's a test environment

Take a look at the config vars currently:

heroku ps -a $APPNAME

There's nothing there right now. When we add our database later Heroku will automatically add a variable DATABASE_URL.

Remember you can use heroku help $COMMAND to find out more about a command:

heroku help config

Cool now go ahead and add these variables:

heroku config:set FLASK_APP=hello -a $APPNAME
heroku config:set APP_ENV=prod -a $APPNAME
heroku config:set FLASK_APP=hello -a $APPNAME-stage
heroku config:set APP_ENV=stage -a $APPNAME-stage

The FLASK_APP variable allows us to run command-line flask inside the Heroku dynos like heroku run flask. We'll be doing that later.

The APP_ENV var allows our app to know which environment it is running in. We are going to use this for our example of how the variables work.

So let's prove it works by simply getting the value of that variable and adding it to our page. Edit your hello/__init__.py to be like this. changes

import os
from flask import Flask
app = Flask(__name__)

@app.route('/')
def index():
    return '<h1>Hello World! This environment is "' \
        + os.environ['APP_ENV'] + '"</h1>'

Commit and push that to both environments (prod and stage).

Test that it works. Hooray!

Now you will wonder how we set variables for running our app locally. Well basically we make a ".env" file. Make it like this:

pip install python-dotenv
echo "FLASK_APP=hello" >> .env
echo "APP_ENV=local" >> .env
heroku local

You don't need to commit that as it's in our .gitignore.

Now I invite you to add another task to your TODO-list for later: read the Flask docs on configuration handling.

We'll be using Postgresql as it's probably the best relational database and we can get one on a free plan from Heroku.

Firstly we're gonna install Postgres locally:

• bookmark the Heroku doc on local postgres setup
• follow the procedure documented at this link to setup Postgres with peer-authentication

Now you've done that it is time to create the database in Heroku. Firstly observe that the postgres addon is not already installed.

heroku addons

See what versions are available in the Heroku docs here, and choose the version that is closest the the version you have installed locally.

Choose what Plan you will be using here. We will use hobby-dev for this "Hello World" because it is free.

heroku addons:create heroku-postgresql:hobby-dev -a $APPNAME --version=13
heroku addons:create heroku-postgresql:hobby-dev -a $APPNAME-stage --version=13

Great now we've made all our databases! Take a look how everything is:

heroku addons
heroku pg:info -a $APPNAME
heroku config -a $APPNAME
heroku pg:credentials:url -a $APPNAME

It's important to learn how to move databases around in Heroku.

Now let's connect to the database and add some dummy data:

heroku pg:psql -a $APPNAME

Now add a table with some dummy data in it:

CREATE TABLE farm (id INT, animal VARCHAR(255));
INSERT INTO farm VALUES (1, 'cow');
SELECT * FROM farm;

Alright quit out of that. Now we will pull that database into the staging environment. This requires getting the details first:

heroku config:get DATABASE_URL -a $APPNAME-stage

OK good now you know the deets please use them in this command:

DEETS=
heroku pg:pull DATABASE_URL $DEETS -a $APPNAME

Of course you can connect to check that worked by heroku pg:psql -a $APPNAME-stage and then \d farm

Now we will pull that down to a database on the local Postgres installation.

Decide on a name for your local database and use it in the below command. DATABASE_URL will automatically refer to your app's live database because you only have one.

LOCALDBNAME=
heroku pg:pull DATABASE_URL $LOCALDBNAME -a $APPNAME

Test with psql $LOCALDBNAME and then \d farm. Cool it worked!

How will the app know the credentials to connect to the database when it starts? Well it gets the settings from the environment variables. Take a look and see:

heroku config -a $APPNAME

For running it locally it gets it from your file ".env". That is the file where we can set variables for our local testing. So update .env with the setting for connecting to your database:

echo "DATABASE_URL=postgresql:///$LOCALDBNAME" >> .env

OK so let me explain the tools we will be using.

• SQLAlchemy is a library that provides tools for accessing a database
• Flask-SQLAlchemy is a wrapper for SQLAlchemy with some extra features which are useful for web-applications
• Alembic is a database migrations tool. You need this because future versions of your software will likely have different database structures and you need a way to upgrade from the old format to the new
• Flask-Migrate is a wrapper for Alembic which makes it easier to integrate into a Flask app

Install flask-sqlalchemy and flask-migrate (these will pull in SQLAlchemy and Alembic as deps):

pip install wheel  # it fails to build alembic without this
pip install flask-sqlalchemy flask-migrate

We also need psycopg2 which requries extra deps to build:

sudo apt install python3-dev libpq-dev build-essential
pip install psycopg2

Add to requirements:

pip freeze | tee requirements.txt

Notice your flask shell command now has an extra command for db migrations:

flask --help

We setup Flask-Migrate before we even make our app use the database. This means it's managing our data schema right from the outset.

But before we do anything please delete all the test data you made earlier as we want Flask-migrate to build up from an empty database:

heroku pg:reset DATABASE_URL -a $APPNAME
heroku pg:reset DATABASE_URL -a $APPNAME-stage
dropdb $LOCALDBNAME  # try dropdb --help for more info
createdb $LOCALDBNAME

OK cool. Now modify your app like this. changes

import os
import re
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = re.sub('postgres://', 'postgresql://', os.environ['DATABASE_URL'])

db = SQLAlchemy(app)
migrate = Migrate(app, db)

class Farm(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    animal = db.Column(db.String(128))
    diet = db.Column(db.String(128))

@app.route('/')
def index():
    return '<h1>Hello World! This environment is "' \
        + os.environ['APP_ENV'] + '"</h1>'

The most interesting part we've just added is the db model. This defines the schema of our dummy database we made earlier, except we've added an extra column.

Now we will initialise flask_migrate:

flask db init

OK now we get to see the power of Alembic. The following command will compare the database structure we defined in our Python code with what exists in Postgres and produce a migration plan:

flask db migrate -m "Initial migration."

Alright now review the migration-plan it just made. For me I have an migrations/versions/f8747a559443_initial_migration.py containing an upgrade function like this:

def upgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    op.create_table('farm',
    sa.Column('id', sa.Integer(), nullable=False),
    sa.Column('animal', sa.String(length=128), nullable=True),
    sa.Column('diet', sa.String(length=128), nullable=True),
    sa.PrimaryKeyConstraint('id')
    )
    # ### end Alembic commands ###

As you can see it is creating database structures to support the db model we defined in our python code.

n.b. you must always review the migration plan as it won't always get it right!

Now we get to upgrade our db! Remember as we are running it locally it gets the database details from our .env file.

flask db upgrade

Awesome! Now you may use psql to connect to your db and check that it added our table:

psql $LOCALDBNAME
\d farm

See that we now also have a table alembic_version

SELECT * FROM alembic_version;

When you create future migrations it uses this to find which migration script to run next. See the Revises field in the migraton plan.

Alright so how do we run the migration on Heroku? Well firstly add the migrations directory to version control and push to our staging environment:

git add migrations
git commit -a
git push stage main:main

OK great now using heroku run we can easily run the migration in our staging environment.

heroku run flask db upgrade -a $APPNAME-stage

Now check that it created the table!

heroku pg:psql -a $APPNAME-stage
\d farm

Great! Now we know it works; repeat that on your prod environment too (from git push).

I recommend you bookmark the docs for Flask-Migrate for future reference.

OK now to update our app to actually use our database.

Remember in your hello/__init__.py you defined your database model like this:

class Farm(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    animal = db.Column(db.String(128))
    diet = db.Column(db.String(128))

Just to recap; we are using SQLAlchemy ORM functionality through a wrapper called Flask-SQLAlchemy. It's wize to bookmark the docs for Flask-SQLAlchemy.

So the next part is to use Python code to write and read the table. Notes on how we do it:

• we will be processing form input using flask "request" object.
• we ought to be using Jinja templates for HTML but we'll introduce that later

OK so modify your hello/__init__.py to be like this: changes

import os
import re
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask import request
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = re.sub('postgres://', 'postgresql://', os.environ['DATABASE_URL'])

db = SQLAlchemy(app)
migrate = Migrate(app, db)

class Farm(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    animal = db.Column(db.String(128))
    diet = db.Column(db.String(128))

@app.route('/', methods=['GET', 'POST'])
def index():
    response = '<h1>Hello World! This environment is "' \
        + os.environ['APP_ENV'] + '"</h1>'

    if request.method == 'POST':
        new_animal = Farm(animal=request.form['animal'], diet=request.form['diet'])
        db.session.add(new_animal)
        db.session.commit()

    response += '<p>The farm has these animals:</p><ul>'
    for entry in Farm.query:
        response += '<li>' + entry.animal + ' eats ' + entry.diet + '</li>'

    response += '</ul>'
    
    response += '''<form action="/" method="post"><br>
Animal: <input type="text" name="animal">
eats: <input type="text" name="diet">
<input type="submit" value="Submit">
</form>'''
    
    return response

Alright now give that a whirl using heroku local. You can add your animal and it's diet and click "Submit". Then the animal is added to the database, and displayed back on the page. Add a few animals just so you have some data.

Commit it to git and promote it to your staging environment to test it works there too.

Now I recommend you add the flask documentation on db models to your TODO-list to ready later: Declaring Models – Flask-SQLAlchemy Documentation

Firstly I want you to break your app by misusing the "Farm" object. Add this to your index() function:

test = Farm['animal']

Now test that using heroku local. The web-page itself does not display any debugging info just a generic error (this is good for security). But you will notice that heroku local displays a Traceback on your terminal.

Now commit and push your broken code to your stage environment.

Try loading the page to produce the error and then see it using

heroku logs -a $APPNAME-stage

If you want heroku local to output some basic info for every request you can set it to run in verbose mode.

Edit your "Procfile" thus:

echo 'web: gunicorn hello:app --log-level DEBUG' > Procfile

Alright cool. So we can see from our traceback that the pertinent error is "TypeError: 'DefaultMeta' object is not subscriptable". But how to fix it? You could look in the official docs for Flask-SQLAlchemy, but Python actually has a built-in help system which can give you specific information much more quickly. Here's how.

Firstly, in your project directory with your virtualenv still active, run python3 interactively. Now run these commands:

from dotenv import load_dotenv
load_dotenv()

Now your python has read your .env and has the same variables as your application does when run locally. Now you can import your Farm object and find out more about it:

from hello import Farm
help(Farm)
help(Farm.query)

\o/ OK good quit out of that. And please remove the broken line you added.

Now to make this easier in future let's add the dotenv import to our app so you wont have to do that step manually. Here's how __init__.py will look now: changes

import os
import re
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask import request
from dotenv import load_dotenv
load_dotenv()

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = re.sub('postgres://', 'postgresql://', os.environ['DATABASE_URL'])

db = SQLAlchemy(app)
migrate = Migrate(app, db)

class Farm(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    animal = db.Column(db.String(128))
    diet = db.Column(db.String(128))

@app.route('/', methods=['GET', 'POST'])
def index():
    response = '<h1>Hello World! This environment is "' \
        + os.environ['APP_ENV'] + '"</h1>'

    if request.method == 'POST':
        new_animal = Farm(animal=request.form['animal'], diet=request.form['diet'])
        db.session.add(new_animal)
        db.session.commit()

    response += '<p>The farm has these animals:</p><ul>'
    for entry in Farm.query:
        response += '<li>' + entry.animal + ' eats ' + entry.diet + '</li>'

    response += '</ul>'
    
    response += '''<form action="/" method="post"><br>
Animal: <input type="text" name="animal">
eats: <input type="text" name="diet">
<input type="submit" value="Submit">
</form>'''
    
    return response

Commit that.

Now I just want to teach you how to send extra messages to the logs.

Basically as documented here you can log extra messages just by using app.logger.

Add this to your index() function:

app.logger.error('Would you love a monsterman?')

Now when you look in your logs in heruko local (or heroku logs once you deploy it) you can see the message.

Now like before I want you to take a look at the help for app.logger. Start a python interactive shell and:

from hello import app
help(app.logger)

Alright cool that worked now you can remove the logging line from your app.

hello/__init__.py

import os
import re
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
from flask import request
from flask import render_template
from dotenv import load_dotenv
load_dotenv()

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = re.sub('postgres://', 'postgresql://', os.environ['DATABASE_URL'])

db = SQLAlchemy(app)
migrate = Migrate(app, db)

class Farm(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    animal = db.Column(db.String(128))
    diet = db.Column(db.String(128))

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        new_animal = Farm(animal=request.form['animal'], diet=request.form['diet'])
        db.session.add(new_animal)
        db.session.commit()

    response = render_template('index.html',
                               environment=os.environ['APP_ENV'],
                               animals=Farm.query
    )
        
    return response

hello/templates/base.html

<!doctype html>
  
<html lang="en">
  <head>
    <meta charset="utf-8">
  
    <title>My Farm</title>
    <meta name="description" content="What the animals on the farm eat.">
  
    <!-- Bootstrap CSS -->
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-eOJMYsd53ii+scO/bJGFsiCZc+5NDVN2yr8+0RDqr0Ql0h+rP48ckxlpbzKgwra6" crossorigin="anonymous">
  </head>
  <body>
    <header>
    </header>
    <main class="container">
        {% block content %}
        {% endblock %}
    </main>
    <footer>
    </footer>
      
    <!-- Bootstrap JS bundle -->
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.0.0-beta3/dist/js/bootstrap.bundle.min.js" integrity="sha384-JEW9xMcG8R+pH31jmWH6WWP0WintQrMb4s7ZOdauHnUtxwoG2vI5DkLtS3qm9Ekf" crossorigin="anonymous"></script>
  </body>
</html>

hello/templates/index.html

{% extends "base.html" %}
{%- block content %}
      <h1>Hello World! This environment is "{{ environment }}"</h1>
      <p>The farm has these animals:</p>
      <ul class="list-group list-group-flush mb-4">
        {%- for entry in animals %}
          <li class="list-group-item">{{ entry.animal }} eats {{ entry.diet }}</li>
        {%- endfor %}
      </ul>
      <form action="/" method="post">
        <div class="mb-3">
          <label for="animal1" class="form-label">Animal</label>
          <input type="text" name="animal" class="form-control" id="animal1">
        </div>
        <div class="mb-3">
          <label for="diet1" class="form-label">eats</label>
          <input type="text" name="diet" class="form-control" id="diet1">
        </div>
        <button type="submit" class="btn btn-primary">Submit</button>
      </form>
{%- endblock %}

Run these commands

flask db init
flask db migrate -m "Initial migration."
flask db upgrade
heroku local  # it works \o/

git add .
git commit
git branch -M main
git push stage main:main
heroku run flask db upgrade -a $APPNAME-stage
git push prod main:main
heroku run flask db upgrade -a $APPNAME

Now visit both sites and check they work.

The project we made in this tutorial is in Gitlab here.

Just a reminder of what your workflow should be when you develop your app.

So you want to add your friend so they can contribute too?

First setup a repository on GitHub or Gitlab as remote origin for your code and push it. Then send your friend a Heroku invite using the docs.

Now your friend needs to:

Throughout this article there were several places where I suggested things to bookmark, or extra reading "for later". Here's a recap:

• I advised you to bookmark the Heroku documentation
• I advised you to read the Flask docs on project layout
• I suggested you might want to eventually consider Heroku Pipelines
• I suggested you read the Flask docs on configuration handling
• I recommended you bookmark the docs for Flask-Migrate
• I recommended you bookmark the docs for Flask-SQLAlchemy
• I advised you to read the Flask-SQLAlchemy docs on declaring models
• I advised you to read the Jinja2 docs
• I linked you to the Bootstrap Docs
• I suggested you look at the Bootstrap examples and templates

I hope this article has given you enough knowledge to get started with Python web development. If you really want to learn more then the O'REILLY book on "Flask Web Development" is probably a good start.