Create a Python CLI Cookiecutter from scratch

Lazy Dev School published the official guide titled “Create a Battle Tested Python3 CLI Project”. Today we are going to convert that project into a template that we can use to start all our future python cli projects with.

Prerequisites

• Python CLI project (the python-cli project from the previous tutorial) LDSOG
• Python 3.10+ with Pipenv virtual environment manager LDSOG
• Git LDSOG

Create a working directory

I know this seems weird but we need a container (this directory) that we put our entire proejct we want to forge into a cookiecutter.

mkdir python_cli_cookiecutter

Change into the new directory and create a file named cookiecutter.json. The cookiecutter.json file is the configuration file for the cookiecutter. It is where we define the variables that we want to use in the template. If that makes no sense, just hang tight. You will fully understand in a few minutes.

cd python_cli_cookiecutter
touch cookiecutter.json

Get a fresh copy of the python-cli project

The CLI project we are converting to a cookiecutter as built in the Official Lazy Dev School tutorial titled “Create a Battle Tested Python3 CLI Project”.

If you have not completed that tutorial, you can go try it out OR just get a shinny new copy of the project from github.

git clone git@github.com:lazydevschool/python_cli.git

Directory Checkpoint Numero Uno

➜  python_cli_cookiecutter tree .
.
├── cookiecutter.json
└── python_cli
    ├── Pipfile
    ├── Pipfile.lock
    ├── README.md
    ├── pytest.ini
    ├── settings.toml
    ├── setup.py
    ├── src
    │   └── pycli
    │       ├── cli.py
    │       ├── config.py
    │       └── main.py
    ├── template.secrets.toml
    └── tests
        └── test_config.py

5 directories, 12 files

Quick Housekeeping

The project that we cloned from github includes a .git directory. We gotta send that to the basura.

Change your directory into the cloned repo and remove .git directory.

cd python_cli
rm -rf .git
cd ..

We do this because our source control tracking needs to handle all the contents of the cookiecutter project, not just the original code.

Initialize git repo for cookiecutter

In the root directory of the cookiecutter project initialize a git repository.

git init

Add initial commit

git add .
git commit -m "initial commit"

While we are at it lets create a new remote repository on GitHub named python_cli_cookiecutter and push our work.

• create new repo with the name python_cli_cookiecutter
• select public / private
• select create repository
• follow your unique instructions for “…or push an existing repository from the command line”

README.md

Dang! We got all trigger happy and made our initial commit wouthout a README.md file. Lets add one now.

echo "# python_cli_cookiecutter" > README.md

git add README.md
git commit -m "add README.md"
git push

Create a feature branch

Let’s create a new branch and name it feature/cookify. Why feature/cookify?

The idea here is that we are going to make a short lived branch to do work. Instead of committing directly to production (likely titled main if you are following the tutorial), we commit to a feature branch and then create a “Pull Request” on GitHub to merge our changes into production. This is a technique called “trunk based development”.

If this is all clear as mud, just follow along and everything will be revealed to you.

Ok let’s get this party started and make a new branch followed by an initial push to GitHub.

git checkout -b feature/cookify && git push -u origin feature/cookify

Update cookiecutter variables

Now is the time I move from command line to code editor. Hop into your favorite editor and open it in the project root.

Look for the file named cookiecutter.json.

The cookiecutter.json file is where we define the variables that we want to use in the template. Update your file with the following.

{
  "project_name": "Battle Tested Python CLI Starter",
  "project_snake": "{{ cookiecutter.project_name | slugify(separator='_') }}",
  "cli_entrypoint": "pcmd"
}

Parametrize directory struture

We need to sprinkle our placeholders around our project. First lets update the directory names we need to parametrize.

Rename the package pycli in python_cli/src/pycli to {{cookiecutter.project_snake}}. From the root of your cookiecutter project run the following.

cd python_cli/src
mv pycli "{{ cookiecutter.project_snake }}"
cd ../..

Next we need to parametrize the project directory. Lets update python_cli to {{cookiecutter.project_snake}}

mv python_cli "{{ cookiecutter.project_snake }}"

Directory Checkpoint Numero Dos

➜  python_cli_cookiecutter git:(feature/cookify) ✗ tree .
.
├── README.md
├── cookiecutter.json
└── {{ cookiecutter.project_snake }}
    ├── Pipfile
    ├── Pipfile.lock
    ├── README.md
    ├── pytest.ini
    ├── settings.toml
    ├── setup.py
    ├── src
    │   └── {{ cookiecutter.project_snake }}
    │       ├── cli.py
    │       ├── config.py
    │       └── main.py
    ├── template.secrets.toml
    └── tests
        └── test_config.py

5 directories, 13 files

Parametrize the cookiecutter’s README.md file.

Despite README.md being one of the more trivial files required to make this final code execute, it has alot of templateing variables in it. I count 3, do you?

Also, verify you have placeholders in your cookiecutter.json file for all the usages we created in the README.md file.

# {{ cookiecutter.project_name }}

- rename template.secrets.toml to .secrets.toml
- pipenv install --dev
- pipenv run pytest
- pipenv run black src/{{ cookiecutter.project_snake }}
- pipenv run {{ cookiecutter.cli_entrypoint }} --help

Parametrize setup.py

I count 3 template variables in the setup.py file. Do you see them?

from setuptools import setup, find_packages

setup(
    name="{{ cookiecutter.project_snake }}",
    version="0.1.0",
    packages=find_packages(where="src"),
    package_dir={"": "src"},
    include_package_data=True,
    entry_points={
        'console_scripts': [
            "{{ cookiecutter.cli_entrypoint }}={{ cookiecutter.project_snake }}.cli:cli",
        ],
    }
)

Parametrize Pipfile

I count 1 template variables in the Pipfile file. Do you see it?

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
click = "*"
dynaconf = "*"
{{ cookiecutter.project_snake }} = {file = ".", editable = true}

[dev-packages]
black = "*"
pytest = "*"

Parametrize tests/test_config.py

I count one template variables in the tests/test_config.py file. I think you are getting the hang of this by now!

import pathlib
from {{ cookiecutter.project_snake }}.config import settings


def test_project_root():
    assert isinstance(settings.PROJECT_ROOT, pathlib.Path)
    assert settings.PROJECT_ROOT.exists()

def test_timezone():
    assert settings.TIMEZONE == "UTC"

def test_api_key():
    assert settings.API_KEY == "secret1234567890"

Parametrize src/{{ cookiecutter.project_snake }}/cli.py

Here we are importing the main function from our parametrized namespace. Pretty cool huh?

import click
from {{ cookiecutter.project_snake }} import main


@click.group()
def cli():
    pass


@cli.command()
@click.option("--name", "-n", required=True, help="Name to greet")
def greeting(name):
    """A greeting for you, but your name is required."""
    main.greeting(name)

Test the cookiecutter

Change direcotry into path containing the cookiecutter project. In my case i need to move up to projects/ becuase projects/python-cli-cookiecutter is where the cookiecutter resides.

If you are currently in the cookiecutter project, you would run this command.

cd ..

We need to install the cookiecutter package. It does not make sense to use a virtual environment in this case because we are not developing the cookiecutter package. What we are doing is invoking the cookiecutter package upon our shiney new project template python-cli-cookiecutter.

For situations like this i prefer to use pipx to install global packages. It’s the clean way to install a package globally.

python -m pip install --upgrade pip

install pipx

pip install pipx && pipx ensurepath

install cookiecutter

pipx install cookiecutter

Now we can invoke the cookiecutter package with our new project template. If you want take a look at cookiecutter on pypi.org

cookiecutter python_cli_cookiecutter

Let’s try something crazy here for our project name to make sure we get handle all kinds of edge cases.

"Python  --  4.12  (Python has version 4? NO!!) cli DemO"

What we want is a slugified version of our project name subsequently snakecased. As you can see we have punctuation, double spaces, double dashes and a mix of upper and lower case.

See how our project_snake is calculated from our project_name? Let’s name the project_snake something reasonable like pycli_demo. Next you will be prompted for the cli entrypoint. Just hit enter to accept the default value pycmd. Once the process is complete change into your new project directory.

Directory Checkpoint Numero Tres

➜  pycli_demo tree .
.
├── Pipfile
├── Pipfile.lock
├── README.md
├── pytest.ini
├── settings.toml
├── setup.py
├── src
│   └── pycli_demo
│       ├── cli.py
│       ├── config.py
│       └── main.py
├── template.secrets.toml
└── tests
    └── test_config.py

4 directories, 11 files

Install Dependencies

Before we can fire this puppy up we need to install the dependencies.

pipenv install --dev

Verify the CLI is working

If you check out the code in the cli.py file you will see we templated an import from the main.py file. Lets see if we can get the CLI working?

First activate the virtual environment.

pipenv shell

Invoke the CLI command.

pcmd greeting --name="Lazy Dev"

You know you are cooking with gas when you see the following.

➜  pycli_demo pipenv run pcmd greeting -n "Lazy Dev"
Hello, Lazy Dev!

Verify the tests are working

The easy work is done. Now we need to get our tests passing. Spoiler alert! They are not going to pass untill we fix some things.

Run the test to see if we can get some clues.

pytest

You will see output about settings object not having an attribute named API_KEY. If we hunt down where API_KEY is referenced we see it is only in our template.secrets.toml file. If we look in our config file we see no reference to template.secrets.toml.

Can you see where this is going? If not read the REAMDE.md file. It tells us to make a copy of the template.secrets.toml file to .secrets.toml.

cp template.secrets.toml .secrets.toml

Lets run our tests again.

pytest

A test is failing still but we get a different error this time.

Pytest is usually pretty helpful in telling us why we fail a test. You can see the expected value and actual value are different. Does it make sense we would not store sensative information in our template file? Sure it does. So in this case the test is expecting the actual secret not a templated secret.

Go into .secrets.toml and change the value of API_KEY to secret1234567890. Run the tests again. Yay! Our tests are passing. Time to commit our changes.

pytest

You should see the following output

tests/test_config.py::test_project_root PASSED
tests/test_config.py::test_timezone PASSED
tests/test_config.py::test_api_key PASSED

Tear Down The Demo

Well congrats we got a cookiecutter working!

Before we hop back over to the cookiecutter to commit changes lets tear down our pycli demo.

# deactivate the virtual environment
deactivate
unset VIRTUAL_ENV
unset PIPENV_ACTIVE

echo $VIRTUAL_ENV
echo $PIPENV_ACTIVE

unset VIRTUAL_ENV
unset PIPENV_ACTIVE

# remve the virtual environment
pipenv --rm

# remove the pycli_demo directory
cd .. && rm -rf pycli_demo

Commit changes and push to GitHub

Hop back into python_cli_cookiecutter directory.

cd python_cli_cookiecutter

Commit our changes to our local repo and push them up to GitHub.

git add .
git commit -m "implemented cookiecutter"
git push

Verify your branch changes were commited

git status

You should see an output such as the following.

➜  python_cli_cookiecutter git:(feature/cookify) git status
On branch feature/cookify
Your branch is up to date with 'origin/feature/cookify'.

nothing to commit, working tree clean

Update the cookiecutter README.md

Oh snap!

We forgot to tell our users how to use the cookiecutter. Let’s update the README.md file in the root of the cookiecutter project. You can put whatever you want into the README.md file. Generally speaking it’s good to show people how to use the project.

Here is a half decent example of what you are aiming for.

# python_cli_cookiecutter

Welcome to the Battle Tested Python CLI Cookiecutter! This project is designed to help you quickly and easily create a well-structured Python CLI application.

## Usage

\`\`\`bash
# using ssh
cookiecutter git@github.com:lazydevschool/python_cli_cookiecutter.git
\`\`\`

OR

Clone the repo to your local machine and run cookiecutter with a path to the directory

\`\`\`bash
cookiecutter /path/to/python-cli-cookiecutter
\`\`\`

You will be prompted for 3 inputs:

- project_name: Human readable project name
- project_snake: Project name in snake case (e.g. my_awesome_project)
- cli_entrypoint: Entrypoint for the CLI (e.g. my_awesome_cli)

:::note

The project snake will be the name you use to do imports in your project so make sure to keep it short, lowercase, and only use underscores (no hyphens, numbers, punctuation, etc.)

:::

Commit and push your changes to GitHub.

git add .
git commit -m "update README.md"
git push

Now if you hop over to GitHub you will see if you are on main branch that the readme looks plain. Hop over to the feature branch and voila! You see your spruced up README.md.

What needs to happen is we need to create a pull request to merge our feature branch into main. Click the Compare and pull request button and follow the prompts.

General Process for Pull Request

• Add a description for pull request: Implemented cookie cutter and tested it end to end on local machine.
• create pull request
• merge pull request
• confirm merge
• optionaly delete feature branch from GitHub

Pull Main Branch to local

The last step in the process here is to pull the main branch down to your local machine.

git checkout main && git pull

You want to do this becuase what if you have more features you want to implement? You would create a new branch titled feature/cookiecutter++ and got to town hacking on it.

Last Last Step

The very last step if verify we did not tell a lie. We said in our README.md file that the user could invoke the cookiecutter directly from a git repo. Let’s verify this is true.

# move up to the directory containing the cookiecutter
cd ..

#invoke the command
cookiecutter git@github.com:lazydevschool/python_cli_cookiecutter.git

DJ Airhorn

Congratulations! You have just created a battle tested Python3 CLI project.