The CI pipeline, job by job¶

Everything that merges into main has passed this pipeline. It is defined in .gitlab-ci.yml and follows one rule throughout: every job is reproducible locally with a single make target that runs the same commands CI runs. If a job fails in CI, run its make target, fix, re-run.

flowchart LR
    lint --> sanity --> units[units] --> build
    build --> docs
    docs --> integration
    integration --> publish
    publish --> pages

Job	Stage	Runs on	Local equivalent
`lint`	lint	every MR, `main`	`make lint`
`sanity` ×2	sanity	every MR, `main`	`make sanity`
`units` ×2	test	every MR, `main`	`make units`
`build`	build	every MR, `main`, tags	`make build`
`docs`	build	every MR, `main`	`make docs-check`
`integration`	integration	manual button on MRs and `main`	`make integration`
`publish`	publish	tags `v*` only	CI only (release step)
`pages`	publish	`main` only	`make docs`

lint¶

Purpose. Keep the Python and YAML in the repository idiomatic and uniform, and catch the cheap-but-real bug classes (unused imports, shadowed variables, mutable default arguments) before a human reviews the MR. This job is the fastest feedback in the pipeline, which is why it runs first.

What it runs.

$ ruff check .                       # pyflakes/pycodestyle/isort/bugbear rule sets (pyproject.toml)
$ ruff format --check .              # formatting is enforced, not suggested
$ ansible-lint --profile production  # strictest ansible-lint profile

ruff replaces the black/isort/flake8 stack v1 used, with the configuration in pyproject.toml. ansible-lint's production profile is the level required of certified collection content; it checks the example playbooks, the integration playbooks, galaxy.yml metadata and changelog presence, not just the plugin code.

When it fails. The output names the rule (for example name[casing]) with a documentation link, and the file/line. Formatting failures do not show a diff - run the local target below and commit the result.

Reproduce locally.

$ make lint     # exactly what CI runs
$ make format   # auto-fix formatting and fixable rules

sanity¶

Purpose. Run ansible-test sanity - ansible-core's own validator suite for collections. Its most important member here is validate-modules, which cross-checks every module's DOCUMENTATION/EXAMPLES/RETURN blocks against the actual argument spec: every option documented, every documented choice really accepted, types matching, version_added sane, GPLv3 license headers present. The v1 collection advertised a state: ACCEPTED value its validator rejected - with this job in the pipeline, that whole class of drift bug cannot merge.

What it runs. The job runs twice, once per supported boundary (this is the same matrix the units job uses - the two together are what backs the support matrix we publish):

Matrix entry	Python	ansible-core
oldest supported	3.11	2.15
current	3.12	2.18

$ ansible-test sanity --requirements --python $PYTHON_VERSION

Besides validate-modules it includes yamllint (over the embedded YAML doc blocks too), import checks (modules must import without side effects), pep8, shebang and ignore-file hygiene checks.

A path quirk worth knowing. ansible-test refuses to run unless the collection sits at .../ansible_collections/netautomate/netorca. CI copies the checkout there first; locally scripts/setup-collection-path.sh does the same into /tmp/netorca-ac (rsync, so re-runs are fast).

When it fails. Output is path/file.py:line:col: error-code: message. The error codes are documented in the ansible-test sanity reference. The usual culprit after editing a module is a DOCUMENTATION block out of sync with the argspec.

Reproduce locally.

$ make sanity
# equivalent to:
$ cd "$(bash scripts/setup-collection-path.sh)" && ansible-test sanity

units¶

Purpose. Prove the module logic without any network or NetOrca instance: the SDK client is fully mocked. The suite (90 tests and growing) covers the shared base layer (exception→failure mapping, environment-variable fallbacks, pagination and limit, filter serialisation), each module's behaviours (ID lookups, empty-not-error semantics, check mode, changed reporting, diffs) and the complete 6×6 change-instance transition matrix - every legal and illegal state transition.

What it runs. Same two-entry matrix as sanity (Python 3.11 + ansible-core 2.15, Python 3.12 + ansible-core 2.18):

$ pytest

Plain pytest works because tests/unit/conftest.py creates a temporary ansible_collections/netautomate/netorca symlink, so the collection imports resolve without any setup on your side.

When it fails. Normal pytest output. Narrow the run while iterating:

$ pytest -k transition          # only the transition-matrix tests
$ pytest tests/unit/plugins/modules/test_netorca_deployed_item.py -x

Reproduce locally. make units. No credentials are ever needed here.

build¶

Purpose. Prove the thing we would ship actually ships: build the Galaxy artifact, install it into a scratch directory, and render the documentation of every module from the installed artifact. This catches packaging-only breakage that no other job sees - galaxy.yml mistakes, build_ignore accidentally excluding a plugin, doc_fragments that resolve from the source tree but not from the package.

What it runs.

$ ansible-galaxy collection build --output-path build/
$ ansible-galaxy collection install build/netautomate-netorca-*.tar.gz -p /tmp/verify-install
$ ANSIBLE_COLLECTIONS_PATH=/tmp/verify-install ansible-doc -l netautomate.netorca --playbook-dir .
$ ANSIBLE_COLLECTIONS_PATH=/tmp/verify-install ansible-doc netautomate.netorca.netorca_service_item_info | head -5

The tarball is kept as a job artifact for 4 weeks - download it from the MR to test an unreleased build (ansible-galaxy collection install <tarball>).

When it fails. Read the first error: either the build itself refused (galaxy.yml), or ansible-doc could not render a module from the installed copy (packaging/fragment problem).

Reproduce locally. make build, then install the tarball somewhere disposable as above.

docs¶

Purpose. Keep the documentation site honest. Two gates: the drift gate - the committed module reference pages must be byte-identical to a fresh render from the modules' DOCUMENTATION blocks (the pages are generated; hand-edits and stale pages both fail) - and the strict site build - mkdocs build --strict fails on any broken link, bad nav entry or missing include.

What it runs.

$ python scripts/generate_module_docs.py --check   # committed pages == fresh render?
$ mkdocs build --strict                            # whole site, warnings are errors

The rendered site is kept as a job artifact - download site/ from the MR and open index.html to preview the documentation exactly as it will publish.

When it fails. The drift gate prints a unified diff and the fix is always the same: run make docs and commit the regenerated pages. Strict-build failures name the file and the broken reference.

Reproduce locally. make docs-check to verify, make docs-serve to browse it live while writing.

integration¶

Purpose. The only job that talks to a real NetOrca instance. It runs the playbooks in tests/integration/ - currently the read-only smoke suite, which exercises every info module and the write modules in check mode, so it is safe by construction against a shared environment.

What it runs.

$ scripts/run-integration.sh

When it runs - and why it is manual. It appears on every MR and main pipeline as a manual play button with allow_failure: true: it needs live credentials (NETORCA_API_URL / NETORCA_API_KEY as masked GitLab CI/CD variables) and depends on the state of a shared sandbox, so it must never block a merge on its own. Without the variables set, the script skips cleanly and the job succeeds - forks and dry runs stay green.

When it fails. Normal ansible-playbook output. First suspects: expired/invalid API key (the failure will carry error_type: auth), or sandbox state drift.

Reproduce locally. Put the credentials in a git-ignored .env (NETORCA_API_URL=..., NETORCA_API_KEY=...) and run make integration. Details in Testing.

publish¶

Purpose. Release to Ansible Galaxy - with a gate that makes sloppy releases impossible. The v1 history contains ~20 manual "release" commits with drifting versions; this job refuses to publish unless the pushed tag and galaxy.yml agree exactly.

What it runs. Only on tags matching v*:

$ VERSION=$(python -c 'import yaml; print(yaml.safe_load(open("galaxy.yml"))["version"])')
$ test "v$VERSION" = "$CI_COMMIT_TAG"     # hard gate: tag == galaxy.yml version
$ ansible-galaxy collection build --output-path build/
$ ansible-galaxy collection publish build/netautomate-netorca-*.tar.gz --token "$GALAXY_API_KEY"

When it fails. Almost always the gate: galaxy.yml version (X) does not match tag (vY). Use scripts/release.sh to cut releases and it cannot happen - see Releasing.

Requirements. The GALAXY_API_KEY CI/CD variable (Galaxy namespace token) must be configured by a maintainer.

pages¶

Purpose. Publish this documentation site. GitLab Pages serves whatever a job literally named pages leaves in public/, so on every merge to main the site at https://ansible.netorca.io/ is rebuilt from the merged docs.

What it runs.

$ python scripts/generate_module_docs.py --check
$ mkdocs build --strict
$ mv site public

When it fails. Same failure modes as the docs job - it is the same build, promoted.

Reproduce locally. make docs builds the identical site into site/.