Create a Python CLI Cookiecutter from scratch
Create a Python CLI Cookiecutter from scratch
Lazy Dev School published the official guide titled “Create a Battle Tested Python3 CLI Project”. Today we are going to convert that project into a template that we can use to start all our future python cli projects with.
If you need any help with the prerequisites, Lazy Dev School has an official tutorial for each of them titled “LDSOG” wich is short for “Lazy Dev School Official Guide”
Prerequisites
- Python CLI project (the
python-cli
project from the previous tutorial) LDSOG - Python 3.10+ with Pipenv virtual environment manager LDSOG
- Git LDSOG
Create a working directory
I know this seems weird but we need a container (this directory) that we put our entire proejct we want to forge into a cookiecutter.
mkdir python_cli_cookiecutter
Change into the new directory and create a file named cookiecutter.json
. The cookiecutter.json
file is the configuration file for the cookiecutter. It is where we define the variables that we want to use in the template. If that makes no sense, just hang tight. You will fully understand in a few minutes.
cd python_cli_cookiecutter
touch cookiecutter.json
Get a fresh copy of the python-cli project
The CLI project we are converting to a cookiecutter as built in the Official Lazy Dev School tutorial titled “Create a Battle Tested Python3 CLI Project”.
If you have not completed that tutorial, you can go try it out OR just get a shinny new copy of the project from github.
git clone git@github.com:lazydevschool/python_cli.git
Directory Checkpoint Numero Uno
➜ python_cli_cookiecutter tree .
.
├── cookiecutter.json
└── python_cli
├── Pipfile
├── Pipfile.lock
├── README.md
├── pytest.ini
├── settings.toml
├── setup.py
├── src
│ └── pycli
│ ├── cli.py
│ ├── config.py
│ └── main.py
├── template.secrets.toml
└── tests
└── test_config.py
5 directories, 12 files
Quick Housekeeping
The project that we cloned from github includes a .git
directory. We gotta send that to the basura.
Change your directory into the cloned repo and remove .git
directory.
cd python_cli
rm -rf .git
cd ..
We do this because our source control tracking needs to handle all the contents of the cookiecutter project, not just the original code.
We are about to do GitHub stuff.
If you are not a GitHub user, now would be a good time to review the Official Lazy Dev School Guide titled”Push and Clone Github Repos with Ed25519 SSH Authentication for Lazy Devs”.
Initialize git repo for cookiecutter
In the root directory of the cookiecutter project initialize a git repository.
git init
Add initial commit
git add .
git commit -m "initial commit"
While we are at it lets create a new remote repository on GitHub named python_cli_cookiecutter
and push our work.
- create new repo with the name
python_cli_cookiecutter
- select public / private
- select create repository
- follow your unique instructions for “…or push an existing repository from the command line”
README.md
Dang! We got all trigger happy and made our initial commit wouthout a README.md
file. Lets add one now.
echo "# python_cli_cookiecutter" > README.md
git add README.md
git commit -m "add README.md"
git push
Create a feature branch
Let’s create a new branch and name it feature/cookify
. Why feature/cookify
?
The idea here is that we are going to make a short lived branch to do work. Instead of committing directly to production (likely titled main
if you are following the tutorial), we commit to a feature branch and then create a “Pull Request” on GitHub to merge our changes into production. This is a technique called “trunk based development”.
If this is all clear as mud, just follow along and everything will be revealed to you.
If you want to learn more about trunk based development checkout the Atlassian article titled “Trunk-based development”
Ok let’s get this party started and make a new branch followed by an initial push to GitHub.
git checkout -b feature/cookify && git push -u origin feature/cookify
Update cookiecutter variables
Now is the time I move from command line to code editor. Hop into your favorite editor and open it in the project root.
Look for the file named cookiecutter.json
.
The cookiecutter.json
file is where we define the variables that we want to use in the template. Update your file with the following.
{
"project_name": "Battle Tested Python CLI Starter",
"project_snake": "{{ cookiecutter.project_name | slugify(separator='_') }}",
"cli_entrypoint": "pcmd"
}
project_name
is the name of the project.project_snake
is the name of the project formatted in snake case.cli_entrypoint
is the name of the command line interface entry point.
Parametrize directory struture
We need to sprinkle our placeholders around our project. First lets update the directory names we need to parametrize.
Rename the package pycli
in python_cli/src/pycli
to {{cookiecutter.project_snake}}
. From the root of your cookiecutter project run the following.
cd python_cli/src
mv pycli "{{ cookiecutter.project_snake }}"
cd ../..
Next we need to parametrize the project directory. Lets update python_cli
to {{cookiecutter.project_snake}}
mv python_cli "{{ cookiecutter.project_snake }}"
Directory Checkpoint Numero Dos
➜ python_cli_cookiecutter git:(feature/cookify) ✗ tree .
.
├── README.md
├── cookiecutter.json
└── {{ cookiecutter.project_snake }}
├── Pipfile
├── Pipfile.lock
├── README.md
├── pytest.ini
├── settings.toml
├── setup.py
├── src
│ └── {{ cookiecutter.project_snake }}
│ ├── cli.py
│ ├── config.py
│ └── main.py
├── template.secrets.toml
└── tests
└── test_config.py
5 directories, 13 files
Parametrize the cookiecutter’s README.md
file.
Make sure you are updating the correct README.md
file. There are two README.md
files in the project. One in the root and one in the {{ cookiecutter.project_snake }}
directory. You want to update the README.md
file in the {{ cookiecutter.project_snake }}
directory.
Despite README.md
being one of the more trivial files required to make this final code execute, it has alot of templateing variables in it. I count 3, do you?
Also, verify you have placeholders in your cookiecutter.json
file for all the usages we created in the README.md
file.
# {{ cookiecutter.project_name }}
- rename template.secrets.toml to .secrets.toml
- pipenv install --dev
- pipenv run pytest
- pipenv run black src/{{ cookiecutter.project_snake }}
- pipenv run {{ cookiecutter.cli_entrypoint }} --help
Parametrize setup.py
I count 3 template variables in the setup.py
file. Do you see them?
from setuptools import setup, find_packages
setup(
name="{{ cookiecutter.project_snake }}",
version="0.1.0",
packages=find_packages(where="src"),
package_dir={"": "src"},
include_package_data=True,
entry_points={
'console_scripts': [
"{{ cookiecutter.cli_entrypoint }}={{ cookiecutter.project_snake }}.cli:cli",
],
}
)
Parametrize Pipfile
I count 1 template variables in the Pipfile
file. Do you see it?
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
click = "*"
dynaconf = "*"
{{ cookiecutter.project_snake }} = {file = ".", editable = true}
[dev-packages]
black = "*"
pytest = "*"
Parametrize tests/test_config.py
I count one template variables in the tests/test_config.py
file. I think you are getting the hang of this by now!
import pathlib
from {{ cookiecutter.project_snake }}.config import settings
def test_project_root():
assert isinstance(settings.PROJECT_ROOT, pathlib.Path)
assert settings.PROJECT_ROOT.exists()
def test_timezone():
assert settings.TIMEZONE == "UTC"
def test_api_key():
assert settings.API_KEY == "secret1234567890"
Parametrize src/{{ cookiecutter.project_snake }}/cli.py
Here we are importing the main function from our parametrized namespace. Pretty cool huh?
import click
from {{ cookiecutter.project_snake }} import main
@click.group()
def cli():
pass
@cli.command()
@click.option("--name", "-n", required=True, help="Name to greet")
def greeting(name):
"""A greeting for you, but your name is required."""
main.greeting(name)
Test the cookiecutter
Change direcotry into path containing the cookiecutter project. In my case i need to move up to projects/ becuase projects/python-cli-cookiecutter is where the cookiecutter resides.
If you are currently in the cookiecutter project, you would run this command.
cd ..
We need to install the cookiecutter package. It does not make sense to use a virtual environment in this case because we are not developing the cookiecutter package. What we are doing is invoking the cookiecutter package upon our shiney new project template python-cli-cookiecutter
.
For situations like this i prefer to use pipx to install global packages. It’s the clean way to install a package globally.
python -m pip install --upgrade pip
install pipx
pip install pipx && pipx ensurepath
install cookiecutter
pipx install cookiecutter
Now we can invoke the cookiecutter package with our new project template. If you want take a look at cookiecutter on pypi.org
cookiecutter python_cli_cookiecutter
Let’s try something crazy here for our project name to make sure we get handle all kinds of edge cases.
"Python -- 4.12 (Python has version 4? NO!!) cli DemO"
What we want is a slugified version of our project name subsequently snakecased. As you can see we have punctuation, double spaces, double dashes and a mix of upper and lower case.
See how our project_snake is calculated from our project_name? Let’s name the project_snake something reasonable like pycli_demo
. Next you will be prompted for the cli entrypoint. Just hit enter to accept the default value pycmd
. Once the process is complete change into your new project directory.
Directory Checkpoint Numero Tres
➜ pycli_demo tree .
.
├── Pipfile
├── Pipfile.lock
├── README.md
├── pytest.ini
├── settings.toml
├── setup.py
├── src
│ └── pycli_demo
│ ├── cli.py
│ ├── config.py
│ └── main.py
├── template.secrets.toml
└── tests
└── test_config.py
4 directories, 11 files
Install Dependencies
Before we can fire this puppy up we need to install the dependencies.
pipenv install --dev
Verify the CLI is working
If you check out the code in the cli.py
file you will see we templated an import from the main.py file. Lets see if we can get the CLI working?
First activate the virtual environment.
pipenv shell
Invoke the CLI command.
pcmd greeting --name="Lazy Dev"
You know you are cooking with gas when you see the following.
➜ pycli_demo pipenv run pcmd greeting -n "Lazy Dev"
Hello, Lazy Dev!
Verify the tests are working
The easy work is done. Now we need to get our tests passing. Spoiler alert! They are not going to pass untill we fix some things.
Run the test to see if we can get some clues.
pytest
You will see output about settings
object not having an attribute named API_KEY
. If we hunt down where API_KEY is referenced we see it is only in our template.secrets.toml
file. If we look in our config file we see no reference to template.secrets.toml
.
Can you see where this is going? If not read the REAMDE.md file. It tells us to make a copy of the template.secrets.toml
file to .secrets.toml
.
cp template.secrets.toml .secrets.toml
Lets run our tests again.
pytest
A test is failing still but we get a different error this time.
Pytest is usually pretty helpful in telling us why we fail a test. You can see the expected value and actual value are different. Does it make sense we would not store sensative information in our template file? Sure it does. So in this case the test is expecting the actual secret not a templated secret.
Go into .secrets.toml
and change the value of API_KEY
to secret1234567890
. Run the tests again.
Yay! Our tests are passing. Time to commit our changes.
pytest
You should see the following output
tests/test_config.py::test_project_root PASSED
tests/test_config.py::test_timezone PASSED
tests/test_config.py::test_api_key PASSED
Tear Down The Demo
Well congrats we got a cookiecutter working!
Before we hop back over to the cookiecutter to commit changes lets tear down our pycli demo.
# deactivate the virtual environment
deactivate
unset VIRTUAL_ENV
unset PIPENV_ACTIVE
Sometimes you can get away with deactivate
but if you start getting weird errors you can see if the environment vairables are set still
echo $VIRTUAL_ENV
echo $PIPENV_ACTIVE
Also, if there are values associated with the environment variables you can unset them.
unset VIRTUAL_ENV
unset PIPENV_ACTIVE
# remve the virtual environment
pipenv --rm
# remove the pycli_demo directory
cd .. && rm -rf pycli_demo
Commit changes and push to GitHub
Hop back into python_cli_cookiecutter
directory.
cd python_cli_cookiecutter
Commit our changes to our local repo and push them up to GitHub.
git add .
git commit -m "implemented cookiecutter"
git push
Verify your branch changes were commited
git status
You should see an output such as the following.
➜ python_cli_cookiecutter git:(feature/cookify) git status
On branch feature/cookify
Your branch is up to date with 'origin/feature/cookify'.
nothing to commit, working tree clean
Update the cookiecutter README.md
Oh snap!
We forgot to tell our users how to use the cookiecutter. Let’s update the README.md
file in the root of the cookiecutter project. You can put whatever you want into the README.md file. Generally speaking it’s good to show people how to use the project.
Here is a half decent example of what you are aiming for.
# python_cli_cookiecutter
Welcome to the Battle Tested Python CLI Cookiecutter! This project is designed to help you quickly and easily create a well-structured Python CLI application.
## Usage
\`\`\`bash
# using ssh
cookiecutter git@github.com:lazydevschool/python_cli_cookiecutter.git
\`\`\`
OR
Clone the repo to your local machine and run cookiecutter with a path to the directory
\`\`\`bash
cookiecutter /path/to/python-cli-cookiecutter
\`\`\`
You will be prompted for 3 inputs:
- project_name: Human readable project name
- project_snake: Project name in snake case (e.g. my_awesome_project)
- cli_entrypoint: Entrypoint for the CLI (e.g. my_awesome_cli)
:::note
The project snake will be the name you use to do imports in your project so make sure to keep it short, lowercase, and only use underscores (no hyphens, numbers, punctuation, etc.)
:::
Once we paste this code into the README.md we need to clean up the escape characters in the code blocks. There should be 3 backticks in a row not 3 with a \
separating backticks.
Commit and push your changes to GitHub.
git add .
git commit -m "update README.md"
git push
Now if you hop over to GitHub you will see if you are on main branch that the readme looks plain. Hop over to the feature branch and voila! You see your spruced up README.md.
What needs to happen is we need to create a pull request to merge our feature branch into main. Click the Compare and pull request
button and follow the prompts.
Watch this process done step by step in this video at 36:45
General Process for Pull Request
- Add a description for pull request:
Implemented cookie cutter and tested it end to end on local machine.
- create pull request
- merge pull request
- confirm merge
- optionaly delete feature branch from GitHub
Pull Main Branch to local
The last step in the process here is to pull the main branch down to your local machine.
git checkout main && git pull
You want to do this becuase what if you have more features you want to implement? You would create a new branch titled feature/cookiecutter++
and got to town hacking on it.
Last Last Step
The very last step if verify we did not tell a lie. We said in our README.md
file that the user could invoke the cookiecutter directly from a git repo. Let’s verify this is true.
# move up to the directory containing the cookiecutter
cd ..
#invoke the command
cookiecutter git@github.com:lazydevschool/python_cli_cookiecutter.git
DJ Airhorn
Congratulations! You have just created a battle tested Python3 CLI project.