In this video we'll demonstrate how to create a scheduled task using the django-crontab package. We will create a task that fetches cryptocurrency data from the CoinMarketCap API and updates the database. We will also demonstrate how to set these cronjobs up within a Docker container that runs our Django application.

The associated video for this post is shown below.

Objectives

In this post, we will:

• Learn how to set up Django within a Docker container.
• Learn how to use the django-crontab package and create scheduled Django tasks.
• Understand how to protect API Keys with the django-environ package.

Project Setup

Starter code is available on the master branch of this repository. Clone this code to a directory on your machine.

In this post, we will make use of Docker - this is required if you are on Windows machines, as the django-crontab package only works with Unix-like systems (Mac, Linux, etc). For Windows users, you must download Docker to follow along - you can get Docker Desktop here.

Note: Docker works on any system, so I recommend that you download Docker to follow along.

Once you have downloaded Docker, you need to create a .env file in the root directory, alongside the Dockerfile and the docker-compose.yml file. This file will store our API Key for CoinMarketCap, which we do not want to commit to the repository. Create an empty .env file for now - we will explain this in more detail later on.

We are going to interact with the CoinMarketCap API, where we can periodically pull cryptocurrency data and update our database via a django-crontab task. We now need to fetch an API Key from CoinMarketCap, which can be done here and is free of charge for basic use.

Once you have an API Key, add this key as a setting in your .env file, as below:

COINMARKETCAP_KEY=esg-fsd-fds-abc-xyz

The key name is COINMARKETCAP_KEY and the value should be set to your own key. We will later reference this environment variable using the django-environ package.

Once the .env file is created, you can run a simple command to get your Django application up and running:

docker-compose up

This command will look at the docker-compose.yml file from the starter code, and will build and run each service that is specified in this file. We will analyze how this works in the following section, but if you do run this command, you should be able to navigate to localhost:8000 and see the running Django application from the starter code!

Docker Setup

Let's take a brief detour to walk through the Dockerfile and the docker-compose.yml file for this project. In a nutshell, Docker allows you to create environments and services that are specific to your applications requirements. For example, for our project, we want a Django application running on a Linux system that allows us to use django-crontab. We can specify this using a Dockerfile.

Docker is widely used because it gives developers and other engineers nice way of setting up environments with minimal effort, regardless of which operating system they may have, as well as which versions of a programming language or framework they might have. Docker also allows for easy application deployments, and works well with DevOps pipelines. It's a valuable skill to have in your repertoire, so let's dive into the basic settings we have in this starter code!

The Dockerfile in our starter code looks as follows:

FROM python:3.8.6-buster

RUN apt update
RUN apt-get install cron -y
RUN alias py=python

ENV PYTHONUNBUFFERED 1

WORKDIR /usr/src/app

COPY ./app .
COPY ./requirements.txt .

RUN pip install -r requirements.txt

# django-crontab logfile
RUN mkdir /cron
RUN touch /cron/django_cron.log

EXPOSE 8000

CMD service cron start && python manage.py runserver 0.0.0.0:8000

On the first line, we define the base Docker image that we are going to extend, which has Python 3.8.6 installed on the Debian 10 Linux platform. To run Django, we need Python, so it's important to have Python installed on our Docker image!

On line 3, we update the package lists from the Linux repositories to get information on the newest versions of packages and their dependencies, and then, on line 4, we install the cron service in our Docker image. We will need cron in order to run the django-crontab tasks.

On line 7, we set an environment variable within our image using the ENV command. To set PYTHONUNBUFFERED to a truthy value will have the effect of sending Python output to the terminal, which is important if you want to see logs and other messages whilst running your Docker container.

On line 9, we set the working directory - any commands that follow this line will use this directory as the current working directory. On lines 11-12, we copy the app directory and the requirements.txt file (both from our starter app) to the working directory. This brings the code from your local filesystem into the container, and from there, we can run the command to install the Python requirements (line 14). This will install Django, as well as other packages we're using such as django-crontab, into the container.

On lines 17-18, we create a directory and within it, create a log file - this will be used to log output from our django-crontab task, later on.

On line 20, we designate that we will be exposing port 8000 from the container, and finally run our CMD that is our default command that will be executed when we start our Docker container. For this, we start up the cron service, and then run the Django development server on port 8000.

This Dockerfile defines our container - it let's us install OS-level software such as cron, and let's us install Python requirements and set up a container for our application. To build the image, you can use the docker build command, and to run the container, you can use the docker run command with multiple arguments - however, we are going to use a simpler method by utilizing a docker-compose.yml file. This file looks as follows:

version: "3.7"

services:
    webapp:
        build: .
        container_name: django-crypto
        volumes:
            - ./app:/usr/src/app
        env_file: .env
        ports:
            - 8000:8000

We define a service called webapp, which builds from the current directory - this means that docker-compose will look for a Dockerfile in the current directory, and will use that to build the image for the webapp. We give the resulting container a name on line 6, and on line 7-8, we mount a volume in our container. This means that any changes we make during development in our ./app directory (which contains the Django code) will be reflected in the working directory of our application, allowing us to make changes without restarting the container each time.

On line 9, we specify an env_file setting, and point to the .env we created earlier. This will load each line in our .env into the environment of the running container, effectively creating environment variables for each of the settings. Finally, we specify the port mappings - we map port 8000 on our machine to port 8000 on the container (where the Django app is running). This will forward any traffic on port 8000 locally into the container at the same port, allowing us to navigate to localhost:8000 and still see the application running within the container.

Normally, the docker-compose.yml file has multiple services in addition to the webapp - for example databases, caches (Redis), Celery workers, React/Vue apps, etc. Here we just have one, but it simplifies running the container - we simply run the command docker-compose up and everything should spin up.

Anyway, that's enough about Docker. Let's get back to the application - we have a running container, and we're now going to add some settings and start building our django-crontab task.

Adding API Key to settings with django-environ

We have an API Key in our .env file, and this is loaded as an environment variable in the running container via the env_file setting in the docker-compose.yml file. We are now going to use django-environ to load this in our settings.py file.

Within settings.py, add the following code:

import environ

# create env object
env = environ.Env()

# CoinMarketCap API url and API key
API_URL = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest"
CMC_API_KEY = env('COINMARKETCAP_KEY')

For more information on how django-environ works, check out this post. We also add an API_URL setting, which points to the API that we are going to get cryptocurrency data from.

We're now going to test this out in the Django Shell, to ensure the environment variables are being properly loaded. Since Django is running in our Docker container, we are going to use the docker exec command to execute the command to open the Django Shell. For more information on this Docker command, check this out.

Run the following command to bring up the shell.

docker exec -ti django-crypto python manage.py shell_plus

Note here, that django-crypto is the name of our running container. Once in the shell, you can import Django's settings and verify that the API Key exists:

print(settings.CMC_API_KEY)

Let's now send a test request to the CoinMarketCap API, with our new API Key attached as an HTTP Header. The API docs specify that a CMC_PRO_API_KEY header should be sent, with the API Key as the value - for more information, check out their API docs. We can use the requests library to test this out in the Django shell.

import requests

# set the header
headers = {
    "X-CMC_PRO_API_KEY": settings.CMC_API_KEY
}

# send the request to the API url and attach the header
resp = requests.get(settings.API_URL, headers=headers)

# convert response to JSON and inspect the data
data = resp.json()['data']

data[0]['name']

If this works, then we're all good to start creating database models to persist the response JSON data!

Creating Cryptocurrency Models

Within our models.py file, add the following models.

from django.db import models

class Crypto(models.Model):
    name = models.CharField(max_length=128)
    symbol = models.CharField(max_length=5)

    def __str__(self):
        return self.name

class CryptoPrice(models.Model):
    crypto = models.ForeignKey(Crypto, on_delete=models.CASCADE, related_name='prices')
    price = models.FloatField()
    timestamp = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"{self.crypto.name} - {self.price}"

The first model Crypto simply stores the names and symbols of the different cryptocurrencies.

The second model CryptoPrices will keep records of the price of a given cryptocurrency at a particular timestamp. When our cron task fetches new data, the prices will be updated with a new timestamp - allowing us to build a historical record of the prices, whilst also having access to the most recent price, for each cryptocurrency in our application.

All the fields on these models can be captured from the API response JSON data. Now that we have the models, let's create the migrations:

docker exec django-crypto python manage.py makemigrations
docker exec django-crypto python manage.py migrate

This will create the tables in our database, within the container. Now, let's create our django-crontab task and put these models to use!

Creating Django-Crontab task

Before we create a task, we need to add the django_crontab app to our INSTALLED_APPS setting within settings.py.

Once this is done, create a file called cron.py within our Django app, and add the following code.

from django.conf import settings
import requests
from core.models import Crypto, CryptoPrice

def fetch_crypto_prices():
    headers = {
        "X-CMC_PRO_API_KEY": settings.CMC_API_KEY
    }

    resp = requests.get(settings.API_URL, headers=headers)

    data = resp.json()['data']

    for currency in data:
        name = currency['name']
        symbol = currency['symbol']
        price = currency['quote']['USD']['price']

        print(name, symbol, price)

        # get_or_create the Crypto object
        crypto = Crypto.objects.get_or_create(name=name, symbol=symbol)[0]

        # now add the latest crypto price
        CryptoPrice.objects.create(crypto=crypto, price=price)

Here, we define a function called fetch_crypto_prices - this function will be run periodically by django-crontab.

The code within the function is similar to the code we executed in the shell, but this time, we are extracting the name and symbol of each cryptocurrency, as well as its price when the cron task is executed. On line 22, we check to see if the given cryptocurrency exists in the database, or otherwise create it, with the .get_or_create() function.

On line 25, we insert the new record with the price/value of the given cryptocurrency when the cron task is run. This means that, when the task is executed by django-crontab, the results will be stored in the database at the given timestamp. Every time the task is run, we get new records for each of the cryptocurrencies, and we add that data to the underlying table.

So now we have the cron function that we want to execute - but how do we configure django-crontab to call this function? For this, we need to add a few things to our settings.py file.

django-crontab settings

In the settings.py file, add the following settings:

CRONJOBS = [
    ('* * * * *', 'core.cron.fetch_crypto_prices', '>> /cron/django_cron.log 2>&1'),
]

# https://pypi.org/project/django-crontab/
CRONTAB_COMMAND_PREFIX = f'COINMARKETCAP_KEY={CMC_API_KEY}'

The CRONJOBS setting is a list that defines all of the cronjobs that you want the django-crontab package to execute. Each element of this list is a tuple, with the first element defining the schedule of the cronjob - this one will run every single minute, but you can define different schedules if you need to (for example: run every Sunday at midnight).

The second element of the tuple points to our function within the cron.py file - this is the function that will be called on the given schedule.

Finally, the third element here sends the output (i.e. the print() statements) within the cron function to the log file that we set up earlier in our Dockerfile. This third element is optional, and can be omitted if you do not need to capture output from the cronjob.

For a handy tool for defining cron schedules, check out this link.

We also have another setting above: CRONTAB_COMMAND_PREFIX . This allows us to prefix our command with additional configuration - as the django-crontab documentation states, this is especially useful for passing environment variables to cronjobs. That's exactly what we do, as we pass our API key to the cronjob.

The last thing we need to do is register our cronjobs with Django - django-crontab provides a management command to add these:

docker exec django-crypto python manage.py crontab add

Once this is done, the task will be called every one minute, and the function within our cron.py file will be executed. You can inspect the log file to see if it is collecting output with the cat command.

docker exec django-crypto cat /cron/django_cron.log

You should, after the first execution of our cron-based function, see output listing the different cryptocurrencies and their prices.

Let's now amend the view and display the database data in a basic manner.

Amending Index View and Template

Let's grab the data for Bitcoin within our index view - change the code in the views.py file to the following.

from django.shortcuts import render
from core.models import Crypto, CryptoPrice

def index(request):
    bitcoin = Crypto.objects.get(name='Bitcoin')
    prices = CryptoPrice.objects.filter(crypto=bitcoin)

    context = {'currency': bitcoin, 'crypto_prices': prices}
    return render(request, 'index.html', context)

We get the Crypto object for Bitcoin, and then filter down the related CryptoPrice objects to only those for Bitcoin. We then render our index.html template with the given context variables.

Within the template's content block, add the following code:

{% block content %} 
    <p>Prices for {{ currency }}</p>

    <ul>
    {% for crypo_price in crypto_prices %}
        <li>{{ crypo_price.price }}</li>
    {% endfor %}
    </ul>
{% endblock %}

We list out each of the different prices we've grabbed for Bitcoin. If we refresh the page AFTER the cronjob has run in the background, we should see a new price added to our list. This is because the cronjob is running periodically in the background, and is adding new prices to our database each time it's invoked.

Bonus: Adding HTMX

Let's quickly add some HTMX polling here, to update the prices without having to refresh. To the <ul> tag, add these HTMX attributes:

<ul hx-get="{% url 'index' %}" hx-trigger="every 60s">

This will poll the view every minute and replace the <ul> tag's innerHTML (i.e. the list) with the returned content. So now we need our view to return a list of prices if it is dealing with an HTMX request. To do this, we'll use the django-htmx package. There are two steps to set this up:

• To the INSTALLED_APPS setting, add: django_htmx
• To the MIDDLEWARE setting, add: django_htmx.middleware.HtmxMiddleware

Now, amend the view with the following code.

from django.http.response import HttpResponse
from django.shortcuts import render
from core.models import Crypto, CryptoPrice

def index(request):
    bitcoin = Crypto.objects.get(name='Bitcoin')
    prices = CryptoPrice.objects.filter(crypto=bitcoin)

    context = {'currency': bitcoin, 'crypto_prices': prices}

    # .htmx property added to Request object by django-htmx package
    if request.htmx:
        # return list elements for each CryptoPrice instance
        return HttpResponse(''.join([f'<li>{p.price}</li>' for p in prices]))
    return render(request, 'index.html', context)

When HTMX polls the backend, the response will be the HTML defined on line 14. You can optionally separate this into its own fragment/partial template, but here we show an alternative approach.

With that change, the page should update every minute with the new data gathered by our cronjob!

Summary

In this post, we have covered a lot of ground, with discussion about Docker and docker-compose, as well as learning about the django-crontab package for defining cron tasks. We defined a function for our cronjob, and learned about the Django settings we need to write in order to configure the cronjob.

We have seen how to use django-environ to protect our API keys, and how to load in a .env file via docker-compose.yml.

If you enjoyed this post, please subscribe to our YouTube channel and follow us on Twitter to keep up with our new content!

Please also consider buying us a coffee, to encourage us to create more posts and videos!