.. workflow:: debian_pipeline

Workflow ``debian_pipeline``
============================
We want to provide a workflow coordinating all the steps that are typically
run to build and test an upload to Debian, similar to the `Salsa CI pipeline
<https://salsa.debian.org/salsa-ci-team/pipeline>`_ but (eventually) with
more distribution-wide testing and the ability to handle the task of
performing the upload.

This builds on the existing :workflow:`sbuild` workflow.

* ``task_data``:

  * ``source_artifact`` (:ref:`lookup-single`, required): the
    :artifact:`debian:source-package` or :artifact:`debian:upload` artifact
    representing the source package to test

  * ``suite`` (:ref:`lookup-single`, optional): a :collection:`debian:suite`
    collection to publish packages to; if set, this suite is also added to
    the start of ``extra_repositories``

  * ``vendor`` (string, required): the distribution vendor on which to run
    tests
  * ``codename`` (string, required): the distribution codename on which to
    run tests
  * ``extra_repositories`` (optional): see :task:`PackageBuild`
  * ``architectures`` (list of strings, optional): if set, only run on any
    of these architecture names

  * ``architectures_allowlist`` (list of strings, optional, either concrete
    architecture names or ``all``): if set, only run on any of these
    architecture names; while ``architectures`` is intended to be supplied
    by users, this field is intended to be provided via
    :ref:`task-configuration`
  * ``architectures_denylist`` (list of strings, optional, either concrete
    architecture names or ``all``): if set, do not run on any of these
    architecture names; this field is intended to be provided via
    :ref:`task-configuration`
  * ``arch_all_build_architecture`` (string, defaults to ``amd64``): concrete
    architecture on which to run tasks for ``Architecture: all`` packages

  * ``signing_template_names`` (dictionary, optional): mapping from
    architecture to list of names of binary packages that should be used as
    signing templates by the :workflow:`make_signed_source` sub-workflow

  * ``sbuild_backend`` (string, optional): see :task:`PackageBuild`
  * ``sbuild_environment_variant`` (string, optional): variant of the
    environment to build on, e.g. ``buildd``

  * ``qa_suite`` (:ref:`lookup-single`, required if
    ``enable_reverse_dependencies_autopkgtest`` or ``enable_debdiff`` is
    True): the :collection:`debian:suite` collection to search for
    reverse-dependencies or debdiff original packages, or to use to generate
    reference QA results to detect regressions. Once we have a good way to
    look up the primary suite for a vendor and codename, this could default
    to doing so
  * ``enable_regression_tracking`` (boolean, defaults to False): configure
    the QA workflow to detect and display regressions in QA results
  * ``regression_tracking_qa_results`` (:ref:`lookup-single`, required if
    ``enable_regression_tracking`` is True): the
    :collection:`debian:qa-results` collection that contains the reference
    results of QA tasks to use to detect regressions
  * ``qa_failure_policy`` (string, defaults to ``ignore``): the policy to
    apply if the ``qa`` workflow fails.  Allowed values are ``ignore``,
    ``fail``, ``confirm``.

  * ``enable_check_installability`` (boolean, defaults to True): whether to
    include installability-checking tasks
  * ``check_installability_suite`` (:ref:`lookup-single`, required if
    ``enable_check_installability`` is True): the :collection:`debian:suite`
    collection to check installability against; once we have a good way to
    look up the primary suite for a vendor and codename, this could default
    to doing so

  * ``enable_autopkgtest`` (boolean, defaults to True): whether to include
    autopkgtest tasks
  * ``autopkgtest_backend`` (string, optional): see :task:`Autopkgtest`

  * ``enable_reverse_dependencies_autopkgtest`` (boolean, defaults to
    False): whether to include autopkgtest tasks for reverse-dependencies

  * ``enable_lintian`` (boolean, defaults to True): whether to include
    lintian tasks
  * ``lintian_backend`` (string, optional): see :task:`Lintian`
  * ``lintian_fail_on_severity`` (string, optional): see :task:`Lintian`

  * ``enable_piuparts`` (boolean, defaults to True): whether to include
    piuparts tasks
  * ``piuparts_backend`` (string, optional): see :task:`Piuparts`
  * ``piuparts_environment`` (string, optional): the environment to run
    piuparts in

  * ``enable_debdiff`` (boolean, defaults to False): whether to include
    debdiff for source and binary packages, comparing the supplied source
    package and the built binary packages against the packages available in the
    distribution identified by ``qa_suite``.

  * ``enable_blhc`` (boolean, defaults to True): whether to include ``blhc``
    tasks for the logs associated to the builds

  * ``enable_confirmation`` (boolean, defaults to False): whether the
    generated workflow includes a confirmation step asking the user to
    double check what was built before the upload

  * ``enable_make_signed_source`` (boolean, defaults to False): whether to
    sign the contents of builds and make a signed source package
  * ``make_signed_source_purpose`` (string, required only if
    ``enable_make_signed_source`` is True): the purpose of the key to sign
    with; see :task:`Sign`
  * ``make_signed_source_key`` (string, required only if
    ``enable_make_signed_source`` is True): the fingerprint to sign
    with; must match ``purpose``

  * ``enable_upload`` (boolean, defaults to False): whether to upload to an
    upload queue
  * ``upload_key`` (:ref:`lookup-single`, optional): key used to sign
    uploads
  * ``upload_binary_key`` (:ref:`lookup-single`, optional): key used to sign
    binary uploads
  * ``upload_require_signature`` (boolean, defaults to True): whether
    uploads must be signed; if this is set and neither ``upload_key`` nor
    (if applicable) ``upload_binary_key`` is set, then the user will have to
    remotely sign the files
  * ``upload_include_source`` (boolean, defaults to True): include
    source with the upload
  * ``upload_include_binaries`` (boolean, defaults to True): include
    binaries with the upload
  * ``upload_merge_uploads`` (boolean, defaults to True): if True, merge the
    uploads for each source package and its binaries that are being
    uploaded, and create one PackageUpload task per source package to upload
    them all together; if False, create a separate PackageUpload task for
    each source and binary upload
  * ``upload_since_version`` (string, optional): if ``source_artifact`` is a
    :artifact:`debian:source-package`, include changelog information from
    all versions strictly later than this version in the ``.changes`` file;
    the default is to include only the topmost changelog entry
  * ``upload_target_distribution`` (string, optional): if
    ``source_artifact`` is a :artifact:`debian:source-package`, override the
    target ``Distribution`` field in the ``.changes`` file to this value;
    the default is to use the distribution from the topmost changelog entry
  * ``upload_target`` (string, defaults to
    ``ftp://anonymous@ftp.upload.debian.org/pub/UploadQueue/``): the upload
    queue, as an ``ftp://`` or ``sftp://`` URL
  * ``upload_delayed_days`` (integer, optional): the number of days to delay
    the upload; this assumes that the upload queue implements Debian's
    convention of uploading delayed uploads to a ``DELAYED/{n}-day`` queue

  * ``publish_replace`` (boolean, defaults to False): if True, replace
    existing similar items, if the target suite allows it

The workflow computes dynamic metadata as:

.. dynamic_data::
  :method: debusine.server.workflows.debian_pipeline::DebianPipelineWorkflow.compute_dynamic_data

The effective set of architectures is ``{architectures}`` (defaulting to all
architectures supported by this Debusine instance and the
``{vendor}:{codename}`` suite, plus ``all``), intersecting
``{architectures_allowlist}`` if set, and subtracting
``{architectures_denylist}`` if set.

The workflow creates sub-workflows and tasks as follows, with substitutions
based on its own task data:

* an :workflow:`sbuild` sub-workflow, with task data:

  * ``input.source_artifact``: ``{source_artifact}``
  * ``target_distribution``: ``{vendor}:{codename}``
  * ``backend``: ``{sbuild_backend}``
  * ``architectures``: the effective set of architectures
  * ``arch_all_build_architecture``: ``{arch_all_build_architecture}``, if set
  * ``environment_variant``: ``{sbuild_environment_variant}``, if set
  * ``signing_template_names``: ``{signing_template_names}``, if set

* if any of ``enable_check_installability``, ``enable_autopkgtest``,
  ``enable_lintian``, ``enable_piuparts`` and ``enable_debdiff`` are True,
  a :workflow:`qa` sub-workflow, with task data copied from the items of the
  same name in this workflow's task data, plus:

  * ``binary_artifacts``:
    ``internal@collections/name:build-{architecture}``, for each available
    architecture
  * ``architectures``: the effective set of architectures

* if ``enable_confirmation`` is set, a :task:`Confirm`

* if ``enable_make_signed_source`` and ``signing_template_names`` are set, a
  :workflow:`make_signed_source` sub-workflow, with task data:

  * ``binary_artifacts``:
    ``internal@collections/name:build-{architecture}``, for each available
    architecture
  * ``signing_template_artifacts``:
    ``internal@collections/name:signing-template-{architecture}-{binary_package_name}``,
    for each architecture and binary package name from
    ``signing_template_names``
  * ``vendor``: ``{vendor}``
  * ``codename``: ``{codename}``
  * ``architectures``: the effective set of architectures
  * ``purpose``: ``{make_signed_source_purpose}``
  * ``key``: ``{make_signed_source_key}``
  * ``sbuild_backend``: ``{sbuild_backend}``

* if ``enable_upload`` is set, a :workflow:`package_upload` sub-workflow for
  each source package being uploaded (at least the top-level
  ``source_artifact``, but also each assembled signed source package from
  the :workflow:`make_signed_source` sub-workflow if one exists), configured
  to require a signature from the developer, with task data:

  * ``source_artifact``: the source artifact to upload (or unset if this
    upload is for the top-level source artifact and
    ``upload_include_source`` is False)
  * ``binary_artifacts``:
    ``internal@collections/name:{prefix}build-{architecture}``, for each
    available architecture (or empty if ``upload_include_binaries`` is
    False), where ``prefix`` is empty if this upload is for the top-level
    source artifact or
    ``signed-source-{architecture}-{binary_package_name}|`` if this upload
    is for an assembled signed source package
  * ``merge_uploads``: ``{upload_merge_uploads}``
  * ``since_version``: ``{upload_since_version}``
  * ``target_distribution``: ``{upload_target_distribution}``
  * ``key``: ``{upload_key}``
  * ``require_signature``: ``{upload_require_signature}``
  * ``target``: ``{upload_target}``
  * ``vendor``: ``{vendor}``
  * ``codename``: ``{codename}``
  * ``arch_all_build_architecture``: ``{arch_all_build_architecture}``
  * ``delayed_days``: ``{upload_delayed_days}``

* if ``suite`` is set, a :workflow:`package_publish` sub-workflow for each
  source package being published (at least the top-level
  ``source_artifact``, but also each assembled signed source package from
  the :workflow:`make_signed_source` sub-workflow if one exists), with task
  data:

  * ``source_artifact``: the source artifact to publish
  * ``binary_artifacts``:
    ``internal@collections/name:{prefix}build-{architecture}``, for each
    available architecture, where ``prefix`` is empty if publishing the
    top-level source artifact or
    ``signed-source-{architecture}-{binary_package_name}|`` if publishing an
    assembled signed source package
  * ``target_suite``: ``{suite}``
  * ``replace``: ``{publish_replace}``

The dependencies between these sub-workflows differ based on the value of
``qa_failure_policy``:

* ``ignore`` (default): The :workflow:`qa` sub-workflow is scheduled but
  nothing depends on it.  If ``enable_confirmation`` is set, then the
  :workflow:`make_signed_source`, :workflow:`package_upload`, and
  :workflow:`package_publish` sub-workflows depend on the :task:`Confirm`
  task.
* ``fail``: The :workflow:`make_signed_source`, :workflow:`package_upload`,
  and :workflow:`package_publish` sub-workflows depend on the :workflow:`qa`
  sub-workflow, so they are not executed before it finishes and are aborted
  if anything fails.
* ``confirm``: The :workflow:`qa` sub-workflow is scheduled with
  ``allow_failure: true`` in its workflow data, and a :task:`Confirm` task
  is scheduled between it and the remaining parts of the workflow.  If the
  :workflow:`qa` sub-workflow succeeds, then that task is confirmed
  automatically; otherwise, it waits for the user to decide whether to
  continue the workflow despite the QA failures.

When ``enable_regression_tracking`` is set
------------------------------------------

* the normal :workflow:`qa` workflow is run with:

  * ``enable_regression_tracking`` set to True
  * ``reference_qa_results`` set with the value of
    ``regression_tracking_qa_results``
  * ``reference_prefix`` set to ``reference-qa-result|``

* an extra :workflow:`qa` workflow is run with:

  * ``enable_regression_tracking`` set to False
  * ``reference_qa_results`` set with the value of
    ``regression_tracking_qa_results``
  * ``update_qa_results`` set to True
  * ``prefix`` set to ``reference-qa-result|``
  * ``source_artifact`` and ``binary_artifacts`` pointing to the
    corresponding artifacts in the :collection:`debian:suite` collection
    listed by ``qa_suite``
  * ``fail_on`` set to ``never``
  * ``enable_debdiff`` set to False (since :workflow:`debdiff` doesn't
    contribute any QA results)

.. todo::

    Not implemented:

    * ``enable_check_installability``, ``check_installability_suite`` (see
      :task:`CheckInstallability`)
