Pytest reporting with html plugin

Hi and welcome back!

In the latest post, we delved into the xdist plugin. This time, we will tip our toes into the HTML plugin and add some integration with test management tools.

Now, why is good reporting crucial in a reliable QA process? In general, QA is a subprocess, a strategic part of the bigger development process, and good reporting ensures transparency and reliability in both the QA team (between manual and automation) and in a broader scope.

How to add pretty reporting:

Let's consider a scenario when you are a young AQA, tasked with adding reporting to your fresh test framework because pytest raw bash output is not readable and pretty enough for business people.

How to solve that? We have lots of different libraries that we could choose from or even craft a solution on our own.

QA is a subprocess, a strategic part of the bigger development process

Allure

One of the existing libraries known to me is Allure, which produces beautiful reports with statistics and other fireworks, smoothly integrating with Jenkins and other CI/CD tools. The main downside of this approach is the added boilerplate code. (While writing this, I found out there is an Allure-pytest plugin too, and I will try to cover it in another post).

Custom solution

We can also craft something on our own (I did it too). The simplest approach is to add a reporting singleton class that, at creation, produces an HTML report header, then adds tests as table rows, and of course, we need to somehow figure out how to get the test status and populate it in the report.

A nice bonus is a place to add screenshots upon failure. Then, in the teardown stage, we need to add closing HTML tags to our report.

The good thing is that this approach is 100% customizable, and depending on your HTML, CSS, JS knowledge, you probably could craft an awe-inspiring report.

Downsides are, of course, development time needed to put into it, testing of the solution, and finally size - how much added code in various places it will take? Will you clutter conftest and tests with additional boilerplate code?

Pytest HTML

The last but not least solution is to use the pytest html plugin that handles some parts of the above for us. The report can be adjusted by using hooks (will dive into it later). The resulting report is, in its basic form, very clean and readable, though I find it lacking.

Implementation:

I must admit I had some ups and downs in that matter.

My former attempt went with appending the docstring value (in pytest-html, there is a mention we can add dosctring and I will show you how), but I think it is against the intended use.

I had also a variable within the test's method '_trid' that worked pretty well, though it was additional boilerplate and apparently it was easy to miss that one.

This is my latest attempt that seems to be clean enough in its purpose.

We start with adding a fixture to a test class:

```python

@pytest.mark.testrail_id("C1231234") def test_navigate_to_elements_page(self, testrail_id):
    """Test method to navigate to the Elements page.""" 
    home_page = ToolsQAHomePage(self) 
    home_page.navigate_to_home_page() 
    elements_page = home_page.navigate_to_elements_page() 
    assert elements_page.is_elements_page_loaded() 
```

In simple terms, a fixture in pytest is a way to set up and provide necessary resources or data for your tests. It allows you to define reusable pieces of code that can be executed before or after your tests.

Now we go to conftest.py

First, we add necessary containers for our data in the HTML report.

```python

def pytest_html_results_table_header(cells):
    cells.insert(3, "<th>TestRail ID</th>") 
    cells.insert(2, "<th>Description</th>") 
   cells.insert(1, '<th class="sortable time" data-column-type="time">Time</th>') 

def pytest_html_results_table_row(report, cells): 
    cells.insert(3, f'<td><a href="http://server/view.html?{report._trid}">{report._trid}</a></td>') 
    cells.insert(2, f"<td>{report.description}</td>") 
    cells.insert(1, f'<td class="col-time">{datetime.utcnow()}</td>')
```

fixture in pytest is a way to set up and provide necessary resources or data for your tests

Next, we add our reporting hook (purposely I made just a slight change to the pre-existing snippet from pytest-html documentation which you can find here):

```python

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield 
    report = outcome.get_result() 
    report.description = str(item.function.__doc__) 
    report._trid = item.get_closest_marker("testrail_id").args[0] 
    extras = getattr(report, "extras", []) 
    if report.when == "call": # always add URL to report
        extras.append(pytest_html.extras.url("http://www.example.com/"))
        xfail = hasattr(report, "wasxfail") 
        if (report.skipped and xfail) or (report.failed and not xfail): 
            # only add additional HTML on failure
            extras.append(pytest_html.extras.html("<div>Additional HTML</div>")) 
report.extras = extras
```

And then comes the tricky part. We need to get the testrail_id value and append it to the report:

```python

@pytest.fixture(scope="session") def testrail_ids_collection(): return []
```

Scope we set for session since we need this data to be persistent. I think in the future would be good to experiment with tweaking it though. The last part we register the fixture in conftest.

```python

# Fixture to associate TestRail IDs with tests
@pytest.fixture def testrail_id(request, testrail_ids_collection):
    tr_id = request.node.get_closest_marker('testrail_id')
    if tr_id: testrail_id_value = tr_id.args[0]
        testrail_ids_collection.append(testrail_id_value) 
        return testrail_id_value 
return None
```

Don't forget to update pytest.ini! We still need to register the marker:

```bash

[pytest] 
markers = testrail_id: associate a TestRail ID with a test
```

And to change our main method to accomodate the html report:

```python
import pytest

if __name__ == '__main__':
    # Set up pytest arguments
    args = [
        '-v',     # Verbose output
        '-s',     # Print output to console
        '-n=2',   # Run tests in parallel using 2 workers
        '--html=report.html',  # HTML test report

    ]

    # Call pytest with arguments
    pytest.main(args)
```

Now when we run our runner or via console we should see something like this:

```bash
plugins: html-4.1.1, metadata-3.0.0, xdist-3.5.0
created: 2/2 workers
2 workers [2 items]

scheduling tests via LoadScheduling

tests/test_navigate_elements_page.py::TestNavigationPage::test_navigate_to_forms_page 
tests/test_navigate_elements_page.py::TestNavigationPage::test_navigate_to_elements_page 
[gw0] PASSED tests/test_navigate_elements_page.py::TestNavigationPage::test_navigate_to_elements_page 
[gw1] PASSED tests/test_navigate_elements_page.py::TestNavigationPage::test_navigate_to_forms_page 

- Generated html report: file:///C:/Users/karlm/PytestProject/PytestProject/report.html -
=================== 2 passed in 10.10s ====================

Process finished with exit code 0
```

good reporting crucial in a reliable QA process

And upon accessing the url:

Summary:

Pytest-html is a powerful plugin that offers a way to generate a concise report that can be handed over to Product Owner, Business Analyst, or shown to other team members.

Its main downside is a bit cumbersome configuration and sadly cluttering of the conftest.py file.

In next post we will try to build integration between our framework and TestRail by outputting from pytest code a JSON file with correct data format. Such file later can be digested by TestRail via use of Postman or simple REST script hidden on the pipeline.

Happy testing!