Testing

Whenever things are built it is important to test them. Testing can be done manually, but this is time consuming and prone to human error. It is better to automate your testing. This will allow you to test your code more frequently and ensure that it is working as expected. For this application unit tests will be used to make sure that all routes function as intended.

Note

These tests should ideally be created as you write your code. This will help you to identify bugs early on and ensure that your code is working as expected.

With our tests we can track the coverage to see if every line of code is tested and if there are any lines that are not tested. 100% coverage can be challenging to achieve as applications grow in complexity, but it is a good goal to aim for.

Tip

No type of testing can guarantee that your code is bug free. It is important to test your code in a variety of ways to ensure that it is working as expected such as unit testing, integration testing, and end-to-end testing.

Unit tests

In Python the pytest framework is commonly used for unit testing. This framework allows you to write tests as functions and run them from the command line. To run the tests you can use the pytest command. This will run all tests in the current directory and subdirectories. You can also specify a specific file or directory to run tests from. The Pytest docs contain a good getting started guide.

Setup

As our tests should run seperate from whatever is happening with our application they will need their own data which means that the tests can always be run.

tests/data.sql

INSERT INTO post (id, title, body)
VALUES (
        1,
        'POST 1',
        'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Aliquam faucibus purus in massa tempor nec feugiat nisl. Diam quis enim lobortis scelerisque fermentum dui. Maecenas sed enim ut sem viverra. Urna condimentum mattis pellentesque id nibh tortor id aliquet. Dignissim sodales ut eu sem integer vitae justo eget magna. Nisi est sit amet facilisis. Aenean euismod elementum nisi quis eleifend. Urna condimentum mattis pellentesque id nibh tortor id aliquet lectus. Urna porttitor rhoncus dolor purus non enim praesent elementum facilisis. Convallis tellus id interdum velit laoreet id donec ultrices. Sed enim ut sem viverra aliquet eget. Mattis enim ut tellus elementum sagittis vitae et leo. In metus vulputate eu scelerisque felis imperdiet. Amet facilisis magna etiam tempor orci.'
    ),
    (
        2,
        'POST 2',
        'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Aliquam faucibus purus in massa tempor nec feugiat nisl. Diam quis enim lobortis scelerisque fermentum dui. Maecenas sed enim ut sem viverra. Urna condimentum mattis pellentesque id nibh tortor id aliquet. Dignissim sodales ut eu sem integer vitae justo eget magna. Nisi est sit amet facilisis. Aenean euismod elementum nisi quis eleifend. Urna condimentum mattis pellentesque id nibh tortor id aliquet lectus. Urna porttitor rhoncus dolor purus non enim praesent elementum facilisis. Convallis tellus id interdum velit laoreet id donec ultrices. Sed enim ut sem viverra aliquet eget. Mattis enim ut tellus elementum sagittis vitae et leo. In metus vulputate eu scelerisque felis imperdiet. Amet facilisis magna etiam tempor orci.'
    ),
    (
        3,
        'POST 3',
        'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Aliquam faucibus purus in massa tempor nec feugiat nisl. Diam quis enim lobortis scelerisque fermentum dui. Maecenas sed enim ut sem viverra. Urna condimentum mattis pellentesque id nibh tortor id aliquet. Dignissim sodales ut eu sem integer vitae justo eget magna. Nisi est sit amet facilisis. Aenean euismod elementum nisi quis eleifend. Urna condimentum mattis pellentesque id nibh tortor id aliquet lectus. Urna porttitor rhoncus dolor purus non enim praesent elementum facilisis. Convallis tellus id interdum velit laoreet id donec ultrices. Sed enim ut sem viverra aliquet eget. Mattis enim ut tellus elementum sagittis vitae et leo. In metus vulputate eu scelerisque felis imperdiet. Amet facilisis magna etiam tempor orci.'
    ),
    (
        4,
        'POST 4',
        'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Aliquam faucibus purus in massa tempor nec feugiat nisl. Diam quis enim lobortis scelerisque fermentum dui. Maecenas sed enim ut sem viverra. Urna condimentum mattis pellentesque id nibh tortor id aliquet. Dignissim sodales ut eu sem integer vitae justo eget magna. Nisi est sit amet facilisis. Aenean euismod elementum nisi quis eleifend. Urna condimentum mattis pellentesque id nibh tortor id aliquet lectus. Urna porttitor rhoncus dolor purus non enim praesent elementum facilisis. Convallis tellus id interdum velit laoreet id donec ultrices. Sed enim ut sem viverra aliquet eget. Mattis enim ut tellus elementum sagittis vitae et leo. In metus vulputate eu scelerisque felis imperdiet. Amet facilisis magna etiam tempor orci.'
    ),
    (
        5,
        'POST 5',
        'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Aliquam faucibus purus in massa tempor nec feugiat nisl. Diam quis enim lobortis scelerisque fermentum dui. Maecenas sed enim ut sem viverra. Urna condimentum mattis pellentesque id nibh tortor id aliquet. Dignissim sodales ut eu sem integer vitae justo eget magna. Nisi est sit amet facilisis. Aenean euismod elementum nisi quis eleifend. Urna condimentum mattis pellentesque id nibh tortor id aliquet lectus. Urna porttitor rhoncus dolor purus non enim praesent elementum facilisis. Convallis tellus id interdum velit laoreet id donec ultrices. Sed enim ut sem viverra aliquet eget. Mattis enim ut tellus elementum sagittis vitae et leo. In metus vulputate eu scelerisque felis imperdiet. Amet facilisis magna etiam tempor orci.'
    );
INSERT INTO comment (body, post_id)
VALUES ('comment 1', 1),
    ('comment 2', 1),
    ('comment 3', 1),
    ('comment 4', 1);

With our data we can create the core config for our tests which will create the temporary database for testing and also share the application to all of the tests to use.

tests/conftest.py

"""Config and setup for tests."""
import os
import tempfile

import pytest
from flaskapp import create_app
from flaskapp.db import get_db, init_db

with open(os.path.join(os.path.dirname(__file__), "data.sql"), "rb") as f:
    _data_sql = f.read().decode("utf8")


@pytest.fixture()
def app():
    """App initialisation."""
    db_fd, db_path = tempfile.mkstemp()

    app = create_app(
        {
            "TESTING": True,
            "DATABASE": db_path,
        },
    )

    with app.app_context():
        init_db()
        get_db().executescript(_data_sql)

    yield app

    os.close(db_fd)
    os.unlink(db_path)


@pytest.fixture()
def client(app):
    """App test client."""
    return app.test_client()


@pytest.fixture()
def runner(app):
    """App cli runner."""
    return app.test_cli_runner()

1. @pytest.fixture can be used in any other test function by including it in a parameter, eg. def test_hello(client): will use the client fixture. This allows us to create predefined configs that can be reused (fixtures can also use each other).

2. The app fixture creates a temporary database and initialises it with the data from data.sql. This is done by creating a temporary file and passing the path to the database to the application. The TESTING config is also set to True to ensure that the application is in testing mode.

We also should test the factory to make sure our app is being created the correct way.

tests/test_factory.py

"""Factory test methods."""
from flaskapp import create_app


def test_config():
    """Test app is in test config mode."""
    assert not create_app().testing
    assert create_app({"TESTING": True}).testing


def test_hello(client):
    """Test hello default route."""
    response = client.get("/hello")
    assert response.data == b"Hello, World!"

Testing the database

Our database should close once the context it is in is closed through the teardown context that was created earlier.

tests/test_db.py

"""Test db functions."""
import sqlite3

import pytest
from flaskapp.db import get_db


def test_get_close_db(app):
    """Test db can be closed."""
    with app.app_context():
        db = get_db()
        assert db is get_db()

    with pytest.raises(sqlite3.ProgrammingError) as e:
        db.execute("SELECT 1")

    assert "closed" in str(e.value)

The create command should also be tested to ensure that it is creating the database correctly. To make sure that the function is being run the function is monkeypatched (modified) to record if it is run.

def test_init_db_command(runner, monkeypatch):
    """Test db init command runs."""

    class Recorder:
        called = False

    def fake_init_db():
        Recorder.called = True

    monkeypatch.setattr("flaskapp.db.init_db", fake_init_db)
    result = runner.invoke(args=["init-db"])
    assert "Initialized" in result.output
    assert Recorder.called

Testing the blog

All the blog routes involve reading from the database so we can test that the correct data is being returned and if data is being correctly added. As we know the contents of the database the tests can be hardcoded for the expected behaviour.

tests/test_blog.py

"""Test blog routes."""
from flaskapp.db import get_db


def test_index(client):
    """Test index page."""
    response = client.get("/")
    assert b"Our Blog" in response.data

    assert b"POST 1" in response.data
    assert b"Continue reading..." in response.data\

def test_post(client):
    """Test post page."""
    response = client.get("/1/post")

    assert b"POST 1" in response.data
    assert (
        b"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod "
        b"tempor incididunt ut labore et dolore magna aliqua. Aliquam faucibus purus "
        b"in massa tempor nec feugiat nisl. Diam quis enim lobortis scelerisque "
        b"fermentum dui. Maecenas sed enim ut sem viverra. Urna condimentum mattis "
        b"pellentesque id nibh tortor id aliquet. Dignissim sodales ut eu sem integer "
        b"vitae justo eget magna. Nisi est sit amet facilisis. Aenean euismod "
        b"elementum nisi quis eleifend. Urna condimentum mattis pellentesque id nibh "
        b"tortor id aliquet lectus. Urna porttitor rhoncus dolor purus non enim "
        b"praesent elementum facilisis. Convallis tellus id interdum velit laoreet id "
        b"donec ultrices. Sed enim ut sem viverra aliquet eget. Mattis enim ut tellus "
        b"elementum sagittis vitae et leo. In metus vulputate eu scelerisque felis "
        b"imperdiet. Amet facilisis magna etiam tempor orci." in response.data
    )


def test_invalid_post(client):
    """Test invalid route for a post."""
    assert client.get("/6/post").status_code == 404


def test_create(client, app):
    """Test creating a post."""
    assert client.get("/create").status_code == 200
    client.post("/create", data={"title": "created", "body": "testing"})

    with app.app_context():
        db = get_db()
        count = db.execute("SELECT COUNT(id) FROM post").fetchone()[0]
        assert count == 6

        post = db.execute("SELECT * FROM post WHERE id=6").fetchone()
        assert post["title"] == "created"
        assert post["body"] == "testing"


def test_invalid_create(client, app):
    """Test invalid create post with no title."""
    assert client.get("/create").status_code == 200
    post = client.post("/create", data={"title": "", "body": "testing"})

    with app.app_context():
        db = get_db()
        count = db.execute("SELECT COUNT(id) FROM post").fetchone()[0]
        assert count == 5

    assert b"Title is required." in post.data


def test_comment(client, app):
    """Test comment view and creation."""
    response = client.get("/1/post")

    assert b"comment 1..." in response.data
    client.post("/1/post", data={"body": "testing comment"})

    with app.app_context():
        db = get_db()
        count = db.execute("SELECT COUNT(id) FROM comment").fetchone()[0]
        assert count == 5

        post = db.execute("SELECT * FROM comment WHERE id=5").fetchone()
        assert post["body"] == "testing comment"


def test_invalid_comment(client, app):
    """Test invalid comment creation with too short of a body."""
    post = client.post("/1/post", data={"body": "short"})

    with app.app_context():
        db = get_db()
        count = db.execute("SELECT COUNT(id) FROM comment").fetchone()[0]
        assert count == 4

    assert b"Comment is too short" in post.data


def test_search(client):
    """Test searching for a post."""
    response = client.get("/search?query=1")

    assert b"POST 1" in response.data

Running the tests

To make it easier to run the tests some of the settings cna be solved in the pyproject.toml file.

[tool.pytest.ini_options]
testpaths = ["tests"]

[tool.coverage.run]
branch = true
source = ["flaskapp"]

The pytest command can now be run which will find and execute all tests. To get a more verbose output the -v flag can be used pytest -v.

$ pytest
============================ test session starts ============================
platform linux -- Python 3.11.1, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/user/Projects/ncea-lvl2-web-flask-example
configfile: pyproject.toml
testpaths: src/tests
collected 12 items

src/tests/test_blog.py ........                                       [ 66%]
src/tests/test_db.py ..                                               [ 83%]
src/tests/test_factory.py ..                                          [100%]

============================ 12 passed in 0.31s =============================

With our tests we can measure the amount of code that is covered with our tests to see if anything is missed.

coverage run -m pytest

Once the tests are run the report can be visualised with coverage report.

$ coverage report
Name                       Stmts   Miss Branch BrPart  Cover
------------------------------------------------------------
src/flaskapp/__init__.py      20      0      6      0   100%
src/flaskapp/blog.py          52      0     22      0   100%
src/flaskapp/db.py            23      0      8      0   100%
------------------------------------------------------------
TOTAL                         95      0     36      0   100%

A more in depth report with details on lines missed can be seen from coverage html which will create a htmlcov directory with the report.

Accessibility Testing

While it is important that your site functions correctly, it is also important that it is accessible to all users. This includes users with disabilities, users with slow internet connections, and users with older devices. There are a number of tools that can be used to test your site for accessibility.

Lighthouse

An easy way to test your site is to use the Lighthouse tool built into Chrome. This tool can be accessed by opening the developer tools and clicking on the Lighthouse tab. This tool will run a number of tests on your site and give you a score for each category. It will also give you a list of things you can do to improve your score.