Test all the things!

Ricardo Sanz Ante

DevOps, Drupal and Angular development

FLOSS and weird music

drupal.org/u/tunic

Walkthrough

Walkthrough

Introduction
How to test?
Test all the things
When and where to test
Implementation

Introduction

Why testing?

Peace of mind
Solid functionality
Early error detection and correction
Working with a safety net
Refactor safely
Do not lose your evenings and weekends
Time/cost savings in the medium and long term
Keep the AIs under control

Introduction

Testing in the AI era

Code is easily generated
False sense of security when coding
AIs may introduce bugs or other defects
Risk of false confidence if tests aren't reviewed.

AIs generate test cases easily
AIs speed up repetitive test writing
AIs can also test

A safety net is even more important!

Introduction

Native Drupal tests

Unit tests
Kernel tests
Functional tests
Functional Javascript tests
Nightwatch

Introduction

Test all the things

There much more to test!

Static code analysis
Package management (security and updates)
Acceptance tests
Visual regression tests
Accessibility
Performance, quality and correctness
Internal and external APIs
HTTP headers
Smoke tests
Deployment tests

Introduction

Environments

Local environment
CI
Production environment
External elements on which the system depends

How to test

Production mirror

Advantages

Content to test all functionalities (over time)
Updated content
Real configuration
It is possible to test special cases only happening in production
Relatively easy to implement... ignoring its problems

How to test

Production mirror

Drawbacks

Automated tests need content that doesn't change
Requires data sanitisation
Uses real configuration
Requires connection with production
Huge databases are a pain
Very long setup time with large projects

How to test

Install site and test content

Install site from configuration and load test content

Advantages

No security or privacy issues
Controlled content, without unexpected changes
Controlled configuration
Completely Drupal-oriented approach
Small data sizes
Short and predictable setup time

How to test

Install site and test content

Drawbacks

Functionalities require specific content
Content must be formalised and maintained
Testing production-only errors is harder
It can miss some cases due to lack of content

How to test

Install site and test content

Implementation

During development, export manually created content wih Default Content.
For testing:

Install site and import configuration
Config Split (or configuration override) to tweak env for tests
Import content with Default Content
Run tests!

Test all the things

Static code analysis

Bug detection
Enforce coding standards
Avoid bad practices
Cyclomatic complexity
CSS, PHP, JavaScript, YAML, Composer...

Test all the things

Static code analysis

Tools

PHPStan
phpcs
phpmd
linters (parallel-lint)

phpmd

phpcs

Test all the things

Static code analysis

When?

CI
Pre-commit Git hooks
Manually triggered

Test all the things

Static code analysis

Wrappers

phpqa: manually or CI triggered
Grumphp: triggered by Git pre-commit hook

Test all the things

PHP packages

Packages with security issues
```
composer audit --locked
```
Outdated packages
```
composer outdated --locked --strict
```
Unapplied patches
Uninstalled modules

Test all the things

PHP packages

When?

Manually
CI
Tests status:
- Updatable packages: warning
- Security issues: warning / error

Test all the things

Acceptance tests

Site functionality tests written in natural language (Behaviour Driven Development)

If all pass, the project is complete
A real browser tests the site
A soft browser (Guzzle) tests the site
Natural language allows
- Discussing tests with non-technical staff/client
- Eliminating ambiguity in requirements
- Writing tests before having code

Test all the things

Acceptance tests

Implementation: Behat

BDD tool on PHP
Behat Extension: module to integrate Drupal and Behat
Behat contexts provide actions ready to be used in tests

              When I am on "/some/page"
              When I fill "username" with "someone"
              When I fill "password" with "123456"
              When I press "login"

Tests can use Drupal API
- Data test creation, pre-conditions, etc
- Test assertions

Test all the things

Acceptance tests

Many additional contexts available...
- nuvole/drupal-behat
- Metadrop/behat-contexts
...that offer many Behat steps ready to use
- Content creation
- Cookie acceptance
- Groups module
- Media
- Paragraphs
- Search API
- ...

Test all the things

Visual regression

Visual error detection
Image comparison
BackstopJS
CLI execution
HTML reports
JUnit format reports (CI integration)

Test all the things

Visual regression

Generate reference images
Compare references with screenshots during testing
Restrict comparison area using CSS selectors
Tolerance level for differences
Interaction via Puppeteer or Playwright

Test all the things

Visual regression

Test components or parts, not entire pages
- Smaller screenshots
- Faulty component is easy to spot
- Smaller repository
- Avoid problems with moving parts (dates)
Animations or videos: stop them in onReady and onBefore
Remove cookie banners and similar elements
Use Metadrop/backstopjs-addons

Test all the things

Accessibility

Forgotten and overlooked
Inclusivity
European Accessibility Act: AA compliance mandatory (June 2025)
Review from project start
Fixing everything at the end is overwhelming
Impacts all layers (design, UX, functionality, etc)
Intuition often fails, review early, review often, especially designs

Test all the things

Accessibility

WAVE

Fully featured
Interactive interface
Browser extension
Can be automated with subscription

Test all the things

Accessibility

IBM Equal Access Accessibility Checker

Fully featured
Excellent documentation
Browser extension
Automated with package on GitHub

Test all the things

Accessibility

Pa11y

Command-line tool (can be automated)
FLOSS
Iterates over a list of pages or sitemap
JUnit format reports (CI integration)
Requires Node.js
Configurable error threshold
Our accessibility tool of choice

Test all the things

Correctness and quality

Lighthouse analysis

Performance
Accessibility
Progressive Web Apps
SEO
Unlighthouse
- Uses Lighthouse but tests an entire site
- Auto-discovery or sitemap
- Detects URL patterns to reduce test volume
Can be automated (CI)

Test all the things

Load and performance

Response times
Breaking point
Can be automated to track trends
- A sudden change in the trend indicates something to review
Tools
- ab test: basic and functional
- Apache JMeter: powerful, GUI, reports
- Gatling: powerful, code-based, reports
- ...

Test all the things

Internal and external APIs

Newman runs Postman collections from the console

Internal
- Test API functionalities

External
- Broken external API, broken website
- Detect third-party issues

Test all the things

HTTP headers

Detect important internal problems

Using Newman

Cache issues

```
x-drupal-dynamic-cache: UNCACHEABLE
```
```
cache-control: no-cache
```
...

Security headers
- HSTS, CSP, Permissions-Policy, etc
- CORS, COEP, CORP, COOP, ...
- ...
...

Test all the things

Smoke tests

No catastrophic error

Check that the site is up and running
Check critical functionalities
Must be quick
Should not change the site or generate any content
Detects:
- Corrupt artefact
- Database down
- Network errors
- Uncommon critical errors

Test all the things

Deployment tests

Critical step for a website
Data updates, library changes, new code...
Perform deployment simulation in test environment
Same data as production (or very similar)
Possible dedicated environment for this purpose
Deploy without surprises

When and where

Local development environment

Manually triggered tests

Must be easy to launch
Always ready

Available tests

Static code analysis
Acceptance tests
PHPUnit
Visual regression
Performance and load
API testing
Smoke tests

When and where

Local development environment

Automatically triggered tests

GrumPHP in Git pre-commit hook
Fast
Very fast

Available tests

Static code analysis
Linters: YML, CSS, Composer, etc
Commit message

When and where

Automated testing environment

Verify that an MR/PR doesn't break anything
Post merge

Available tests

Static code analysis
Packages (security and updates)
Smoke tests (and stop if it fails!)
Acceptance tests
PHPUnit
Visual regression
Performance and load (store trend)
API testing
HTTP headers

When and where

Production environment

Test the most critical elements after a deployment
Goodbye to uncertainty and manual team testing
Non-destructive and no content creation tests
NOT a monitoring system

Available tests

Smoke tests
Limited acceptance tests
Limited visual regression
Performance and load (store trend)
API testing
HTTP headers

Implementation

Tools

Behat
BackstopJS
WAVE
IBM Equal Access Accessibility Checker
Pa11y
ab test
Apache JMeter
Gatling
Newman
Composer
PHPStan
Grumphp
phpqa
phpcs
phpmd
phplint
yamllint
jsonlint
parallel-lint

Implementation

GitLab templates

Implementation of Drupal.org tests
GitLab Templates on Drupal.org
Specific for GitLab
Code analysis, PHPUnit, Nightwatch, etc

Implementation

Aljibe

Set of DDEV plugins
- Behat
- BackstopJS
- Unlighthouse
- Static code analysis (PHPStan, phpcs, linters, etc)
- Git integration (GrumPHP)
- Pa11y
- Newman
- Composer (security and updates)
- ...and more!
https://github.com/Metadrop/ddev-aljibe/
Stable, the foundation for our projects
Can be used in CI
Detailed article: Aljibe: quality and testing for Drupal developments with DDEV

Final notes

Test all the things

Static code analysis
Package management (security and updates)
Acceptance tests
Visual regression tests
Accessibility
Performance, quality and correctness
Internal and external APIs
HTTP headers
Smoke tests
Deployment tests

Final notes

Test all the things

Tests must be GREEN!
Automate all automatable tests
Make manual tests easy to execute
Any possible error that can be tested should be tested
Prioritise based on the criticality of the site/functionality
Live peacefully
Have I mentioned that tests must be GREEN?