Lines of Code

LOC, or Lines of Code, is the crudest measure of complexity. It is debatable whether there is any direct correlation between the lines of code and the complexity of an application, but the indirect correlation is clear. After all, a program with 5 lines is likely simpler than one with 5 million.

When looking at Python metrics, we try to ignore blank lines and lines containing comments.

Lines of code can be calculated using the wc command on Linux and Mac OS, where file.py is the name of the file you want to measure:

If you want to add the combined lines in a folder by recursively searching for all .py files, you can combine wc with the find command:

For Windows, PowerShell offers a word count command in Measure-Object and a recursive file search in Get-ChildItem:

In the response, you will see the total number of lines.

Why are lines of code used to quantify the amount of code in your application? The assumption is that a line of code roughly equates to a statement. Lines is a better measure than characters, which would include whitespace.

In Python, we are encouraged to put a single statement on each line. This example is 9 lines of code:

If you used only lines of code as your measure of complexity, it could encourage the wrong behaviors.

Python code should be easy to read and understand. Taking that last example, you could reduce the number of lines of code to 3:

But the result is hard to read, and PEP 8 has guidelines around maximum line length and line breaking. You can check out How to Write Beautiful Python Code With PEP 8 for more on PEP 8.

This code block uses 2 Python language features to make the code shorter:

• Compound statements: using ;
• Chained conditional or ternary statements: name = value if condition else value if condition2 else value2

We have reduced the number of lines of code but violated one of the fundamental laws of Python:

“Readability counts”

— Tim Peters, Zen of Python

This shortened code is potentially harder to maintain because code maintainers are humans, and this short code is harder to read. We will explore some more advanced and useful metrics for complexity.

Cyclomatic Complexity

Cyclomatic complexity is the measure of how many independent code paths there are through your application. A path is a sequence of statements that the interpreter can follow to get to the end of the application.

One way to think of cyclomatic complexity and code paths is imagine your code is like a railway network.

For a journey, you may need to change trains to reach your destination. The Lisbon Metropolitan railway system in Portugal is simple and easy to navigate. The cyclomatic complexity for any trip is equal to the number of lines you need to travel on:

If you needed to get from Alvalade to Anjos, then you would travel 5 stops on the linha verde (green line):

This trip has a cyclomatic complexity of 1 because you only take 1 train. It’s an easy trip. That train is equivalent in this analogy to a code branch.

If you needed to travel from the Aeroporto (airport) to sample the food in the district of Belém, then it’s a more complicated journey. You would have to change trains at Alameda and Cais do Sodré:

This trip has a cyclomatic complexity of 3, because you take 3 trains. You might be better off taking a taxi!

Seeing as how you’re not navigating Lisbon, but rather writing code, the changes of train line become a branch in execution, like an if statement.

Let’s explore this example:

There is only 1 way this code can be executed, so it has a cyclomatic complexity of 1.

If we add a decision, or branch to the code as an if statement, it increases the complexity:

Even though there is only 1 way this code can be executed, as x is a constant, this has a cyclomatic complexity of 2. All of the cyclomatic complexity analyzers will treat an if statement as a branch.

This is also an example of overly complex code. The if statement is useless as x has a fixed value. You could simply refactor this example to the following:

That was a toy example, so let’s explore something a little more real.

main() has a cyclomatic complexity of 5. I’ll comment each branch in the code so you can see where they are:

There are certainly ways that code can be refactored into a far simpler alternative. We’ll get to that later.

In the following examples, we will use the radon library from PyPI to calculate metrics. You can install it now:

To calculate cyclomatic complexity using radon, you can save the example into a file called cyclomatic_example.py and use radon from the command line.

The radon command takes 2 main arguments:

1. The type of analysis (cc for cyclomatic complexity)
2. A path to the file or folder to analyze

Execute the radon command with the cc analysis against the cyclomatic_example.py file. Adding -s will give the cyclomatic complexity in the output:

The output is a little cryptic. Here is what each part means:

• F means function, M means method, and C means class.
• main is the name of the function.
• 4 is the line the function starts on.
• B is the rating from A to F. A is the best grade, meaning the least complexity.
• The number in parentheses, 6, is the cyclomatic complexity of the code.

Halstead Metrics

The Halstead complexity metrics relate to the size of a program’s codebase. They were developed by Maurice H. Halstead in 1977. There are 4 measures in the Halstead equations:

• Operands are values and names of variables.
• Operators are all of the built-in keywords, like if, else, for or while.
• Length (N) is the number of operators plus the number of operands in your program.
• Vocabulary (h) is the number of unique operators plus the number of unique operands in your a program.

There are then 3 additional metrics with those measures:

• Volume (V) represents a product of the length and the vocabulary.
• Difficulty (D) represents a product of half the unique operands and the reuse of operands.
• Effort (E) is the overall metric that is a product of volume and difficulty.

All of this is very abstract, so let’s put it in relative terms:

• The effort of your application is highest if you use a lot of operators and unique operands.
• The effort of your application is lower if you use a few operators and fewer variables.

For the cyclomatic_complexity.py example, operators and operands both occur on the first line:

import is an operator, and sys is the name of the module, so it’s an operand.

In a slightly more complex example, there are a number of operators and operands:

There are 5 operators in this example:

1. if
2. (
3. )
4. >
5. :

Furthermore, there are 2 operands:

1. sys.argv
2. 1

Be aware that radon only counts a subset of operators. For example, parentheses are excluded in any calculations.

To calculate the Halstead measures in radon, you can run the following command:

Maintainability Index

The maintainability index brings the McCabe Cyclomatic Complexity and the Halstead Volume measures in a scale roughly between zero and one-hundred.

If you’re interested, the original equation is as follows:

In the equation, V is the Halstead volume metric, C is the cyclomatic complexity, and L is the number of lines of code.

If you’re as baffled as I was when I first saw this equation, here’s it means: it calculates a scale that includes the number of variables, operations, decision paths, and lines of code.

It is used across many tools and languages, so it’s one of the more standard metrics. However, there are numerous revisions of the equation, so the exact number shouldn’t be taken as fact. radon, wily, and Visual Studio cap the number between 0 and 100.

On the maintainability index scale, all you need to be paying attention to is when your code is getting significantly lower (toward 0). The scale considers anything lower than 25 as hard to maintain, and anything over 75 as easy to maintain. The Maintainability Index is also referred to as MI.

The maintainability index can be used as a measure to get the current maintainability of your application and see if you’re making progress as you refactor it.

To calculate the maintainability index from radon, run the following command:

In this result, A is the grade that radon has applied to the number 87.42 on a scale. On this scale, A is most maintainable and F the least.

Installing wily

wily is available on PyPI and can be installed using pip:

Once wily is installed, you have some commands available in your command-line:

• wily build: iterate through the Git history and analyze the metrics for each file
• wily report: see the historical trend in metrics for a given file or folder
• wily graph: graph a set of metrics in an HTML file

Building a Cache

Before you can use wily, you need to analyze your project. This is done using the wily build command.

For this section of the tutorial, we will analyze the very popular requests package, used for talking to HTTP APIs. Because this project is open-source and available on GitHub, we can easily access and download a copy of the source code:

You will see a number of folders here, for tests, documentation, and configuration. We’re only interested in the source code for the requests Python package, which is in a folder called requests.

Call the wily build command from the cloned source code and provide the name of the source code folder as the first argument:

This will take a few minutes to analyze, depending on how much CPU power your computer has:

Collecting Data on Your Project

Once you have analyzed the requests source code, you can query any file or folder to see key metrics. Earlier in the tutorial, we discussed the following:

• Lines of Code
• Maintainability Index
• Cyclomatic Complexity

Those are the 3 default metrics in wily. To see those metrics for a specific file (such as requests/api.py), run the following command:

wily will print a tabular report on the default metrics for each Git commit in reverse date order. You will see the most recent commit at the top and the oldest at the bottom:

This tells us that the requests/api.py file has:

• 158 lines of code
• A perfect maintainability index of 100
• A cyclomatic complexity of 9

To see other metrics, you first need to know the names of them. You can see this by running the following command:

You will see a list of operators, modules that analyze the code, and the metrics they provide.

To query alternative metrics on the report command, add their names after the filename. You can add as many metrics as you wish. Here’s an example with the Maintainability Rank and the Source Lines of Code:

You will see the table now has 2 different columns with the alternative metrics.

Graphing Metrics

Now that you know the names of the metrics and how to query them on the command line, you can also visualize them in graphs. wily supports HTML and interactive charts with a similar interface as the report command:

Your default browser will open with an interactive chart like this:

You can hover over specific data points, and it will show the Git commit message as well as the data.

If you want to save the HTML file in a folder or repository, you can add the -o flag with the path to a file:

There will now be a file called my_report.html that you can share with others. This command is ideal for team dashboards.

wily as a pre-commit Hook

wily can be configured so that before you commit changes to your project, it can alert you to improvements or degradations in complexity.

wily has a wily diff command, that compares the last indexed data with the current working copy of a file.

To run a wily diff command, provide the names of the files you have changed. For example, if I made some changes to requests/api.py you will see the impact on the metrics by running wily diff with the file path:

In the response, you will see all of the changed metrics, as well as the functions or classes that have changed for cyclomatic complexity:

The diff command can be paired with a tool called pre-commit. pre-commit inserts a hook into your Git configuration that calls a script every time you run the git commit command.

To install pre-commit, you can install from PyPI:

Add the following to a .pre-commit-config.yaml in your projects root directory:

Once setting this, you run the pre-commit install command to finalize things:

Whenever you run the git commit command, it will call wily diff along with the list of files you’ve added to your staged changes.

wily is a useful utility to baseline the complexity of your code and measure the improvements you make when you start to refactor.

Finding Callers and Usages of Functions and Classes

Before you remove a method or class or change the way it behaves, you’ll need to know what code depends on it. PyCharm can search for all usages of a method, function, or class within your project.

To access this feature, select a method, class, or variable by right-clicking and select Find Usages:

All of the code that uses your search criteria is shown in a panel at the bottom. You can double-click on any item to navigate directly to the line in question.

Using the PyCharm Refactoring Tools

Some of the other refactoring commands include the ability to:

• Extract methods, variables, and constants from existing code
• Extract abstract classes from existing class signatures, including the ability to specify abstract methods
• Rename practically anything, from a variable to a method, file, class, or module

Here is an example of renaming the same api.py module you renamed earlier using the rope module to new_api.py:

The rename command is contextualized to the UI, which makes refactoring quick and simple. It has updated the imports automatically in __init__.py with the new module name.

Another useful refactor is the Change Signature command. This can be used to add, remove, or rename arguments to a function or method. It will search for usages and update them for you:

You can set default values and also decide how the refactoring should handle the new arguments.

What Is Boilerplate?

Boilerplate code is code that has to be used in many places with little or no alterations.

Taking our train network as an example, if we were to convert that into types using Python classes and Python 3 type hints, it might look something like this:

Now, you might also want to add other magic methods, like .__eq__(). This code is boilerplate. There’s no business logic or any other functionality here: we’re just copying data from one place to another.

A Case for dataclasses

Introduced into the standard library in Python 3.7, with a backport package for Python 3.6 on PyPI, the dataclasses module can help remove a lot of boilerplate for these types of classes where you’re just storing data.

To convert the Line class above to a dataclass, convert all of the fields to class attributes and ensure they have type annotations:

You can then create an instance of the Line type with the same arguments as before, with the same fields, and even .__str__(), .__repr__(), and .__eq__() are implemented:

Dataclasses are a great way to reduce code with a single import that’s already available in the standard library. For a full walkthrough, you can checkout The Ultimate Guide to Data Classes in Python 3.7.

Some attrs Use Cases

attrs is a third party package that’s been around a lot longer than dataclasses. attrs has a lot more functionality, and it’s available on Python 2.7 and 3.4+.

If you are using Python 3.5 or below, attrs is a great alternative to dataclasses. Also, it provides many more features.

The equivalent dataclasses example in attrs would look similar. Instead of using type annotations, the class attributes are assigned with a value from attrib(). This can take additional arguments, such as default values and callbacks for validating input:

attrs can be a useful package for removing boilerplate code and input validation on data classes.