Tide

• 1: Configuring Tide
• 2: Maintainer's Guide to Tide
• 3: PR Author's Guide to Tide

    graph TD;
        subgraph github[GitHub]
            subgraph org/repo/branch
                head-ref[HEAD ref];
                pullrequest[Pull Request];
                status-context[Status Context];
            end
        end

        subgraph prow-cluster
            prowjobs[Prowjobs];
            config.yaml;
        end

        subgraph tide-workflow
            Tide;
            pools;
            divided-pools;
            pools-->|dividePool|divided-pools;
            filtered-pools;
            subgraph syncSubpool
                pool-i;
                pool-n;
                pool-n1;
                accumulated-batch-prowjobs-->|filter out <br> incorrect refs <br> no longer meet merge requirement|valid-batches;
                valid-batches-->accumulated-batch-success;
                valid-batches-->accumulated-batch-pending;
                
                status-context-->|fake prowjob from context|filtered-prowjobs;
                filtered-prowjobs-->|accumulate|map_context_best-result;
                map_context_best-result-->map_pr_overall-results;
                map_pr_overall-results-->accumulated-success;
                map_pr_overall-results-->accumulated-pending;
                map_pr_overall-results-->accumulated-stale;
                
                subgraph all-accumulated-pools
                    accumulated-batch-success;
                    accumulated-batch-pending;
                    accumulated-success;
                    accumulated-pending;
                    accumulated-stale;
                end

                accumulated-batch-success-..->accumulated-batch-success-exist{Exist};
                accumulated-batch-pending-..->accumulated-batch-pending-exist{Exist};
                accumulated-success-..->accumulated-success-exist{Exist};
                accumulated-pending-..->accumulated-pending-exist{Exist};
                accumulated-stale-..->accumulated-stale-exist{Exist};
                pool-i-..->require-presubmits{Require Presubmits};
                accumulated-batch-success-exist-->|yes|merge-batch[Merge batch];
                merge-batch-->|Merge Pullrequests|pullrequest;
                accumulated-batch-success-exist-->|no|accumulated-batch-pending-exist;
                accumulated-batch-pending-exist-->|no|accumulated-success-exist;
                accumulated-success-exist-->|yes|merge-single[Merge Single];
                merge-single-->|Merge Pullrequests|pullrequest;
                require-presubmits-->|no|wait;
                accumulated-success-exist-->|no|require-presubmits;
                require-presubmits-->|yes|accumulated-pending-exist;
                accumulated-pending-exist-->|no|can-trigger-batch{Can Trigger New Batch};
                can-trigger-batch-->|yes|trigger-batch[Trigger new batch];
                can-trigger-batch-->|no|accumulated-stale-exist;
                accumulated-stale-exist-->|yes|trigger-highest-pr[Trigger Jobs on Highest Priority PR];
                accumulated-stale-exist-->|no|wait;
            end
        end

        Tide-->pools[Pools - grouped PRs, prow jobs by org/repo/branch];
        pullrequest-->pools;
        divided-pools-->|filter out prs <br> failed prow jobs <br> pending non prow checks <br> merge conflict <br> invalid merge method|filtered-pools;
        head-ref-->divided-pools;
        prowjobs-->divided-pools;
        config.yaml-->divided-pools;
        filtered-pools-->pool-i;
        filtered-pools-->pool-n;
        filtered-pools-->pool-n1[pool ...];
        pool-i-->|report tide status|status-context;
        pool-i-->|accumulateBatch|accumulated-batch-prowjobs;
        pool-i-->|accumulateSerial|filtered-prowjobs;



        classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
        classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
        classDef github fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
        classDef pools-def fill:#00ffff,stroke:#bbb,stroke-width:2px,color:#326ce5;
        classDef decision fill:#ffff00,stroke:#bbb,stroke-width:2px,color:#326ce5;
        classDef outcome fill:#00cc66,stroke:#bbb,stroke-width:2px,color:#326ce5;
        class prowjobs,config.yaml k8s;
        class Tide plain;
        class status-context,head-ref,pullrequest github;
        class accumulated-batch-success,accumulated-batch-pending,accumulated-success,accumulated-pending,accumulated-stale pools-def;
        class accumulated-batch-success-exist,accumulated-batch-pending-exist,accumulated-success-exist,accumulated-pending-exist,accumulated-stale-exist,can-trigger-batch,require-presubmits decision;
        class trigger-highest-pr,trigger-batch,merge-single,merge-batch,wait outcome;

1 - Configuring Tide

Configuration of Tide is located under the config/prow/config.yaml file. All configuration for merge behavior and criteria belongs in the tide yaml struct, but it may be necessary to also configure presubmits for Tide to run against PRs (see ‘Configuring Presubmit Jobs’ below).

This document will describe the fields of the tide configuration and how to populate them, but you can also check out the GoDocs for the most up to date configuration specification.

To deploy Tide for your organization or repository, please see how to get started with prow.

General configuration

The following configuration fields are available:

• sync_period: The field specifies how often Tide will sync jobs with GitHub. Defaults to 1m.
• status_update_period: The field specifies how often Tide will update GitHub status contexts. Defaults to the value of sync_period.
• queries: List of queries (described below).
• merge_method: A key/value pair of an org/repo as the key and merge method to override the default method of merge as value. Valid options are squash, rebase, and merge. Defaults to merge.
• merge_commit_template: A mapping from org/repo or org to a set of Go templates to use when creating the title and body of merge commits. Go templates are evaluated with a PullRequest (see PullRequest type). This field and map keys are optional.
• target_urls: A mapping from “*”, , or <org/repo> to the URL for the tide status contexts. The most specific key that matches will be used.
• pr_status_base_urls: A mapping from “*”, , or <org/repo> to the base URL for the PR status page. If specified, this URL is used to construct a link that will be used for the tide status context. It is mutually exclusive with the target_urls field.
• max_goroutines: The maximum number of goroutines spawned inside the component to handle org/repo:branch pools. Defaults to 20. Needs to be a positive number.
• blocker_label: The label used to identify issues which block merges to repository branches.
• squash_label: The label used to ask Tide to use the squash method when merging the labeled PR.
• rebase_label: The label used to ask Tide to use the rebase method when merging the labeled PR.
• merge_label: The label used to ask Tide to use the merge method when merging the labeled PR.

Merge Blocker Issues

Tide supports temporary holds on merging into branches via the blocker_label configuration option. In order to use this option, set the blocker_label configuration option for the Tide deployment. Then, when blocking merges is required, if an open issue is found with the label it will block merges to all branches for the repo. In order to scope the branches which are blocked, add a branch:name token to the issue title. These tokens can be repeated to select multiple branches and the tokens also support quoting, so branch:"name" will block the name branch just as branch:name would.

Queries

The queries field specifies a list of queries. Each query corresponds to a set of open PRs as candidates for merging. It can consist of the following dictionary of fields:

• orgs: List of queried organizations.
• repos: List of queried repositories.
• excludedRepos: List of ignored repositories.
• labels: List of labels any given PR must posses.
• missingLabels: List of labels any given PR must not posses.
• excludedBranches: List of branches that get excluded when querying the repos.
• includedBranches: List of branches that get included when querying the repos.
• author: The author of the PR.
• reviewApprovedRequired: If set, each PR in the query must have at least one approved GitHub pull request review present for merge. Defaults to false.

Under the hood, a query constructed from the fields follows rules described in https://help.github.com/articles/searching-issues-and-pull-requests/. Therefore every query is just a structured definition of a standard GitHub search query which can be used to list mergeable PRs. The field to search token correspondence is based on the following mapping:

• orgs -> org:kubernetes
• repos -> repo:kubernetes/test-infra
• labels -> label:lgtm
• missingLabels -> -label:do-not-merge
• excludedBranches -> -base:dev
• includedBranches -> base:master
• author -> author:batman
• reviewApprovedRequired -> review:approved

Every PR that needs to be rebased or is failing required statuses is filtered from the pool before processing

Context Policy Options

A PR will be merged when all checks are passing. With this option you can customize which contexts are required or optional.

By default, required and optional contexts will be derived from Prow Job Config. This allows to find if required checks are missing from the GitHub combined status.

If branch-protection config is defined, it can be used to know which test needs be passing to merge a PR.

When branch protection is not used, required and optional contexts can be defined globally, or at the org, repo or branch level.

If we want to skip unknown checks (ie checks that are not defined in Prow Config), we can set skip-unknown-contexts to true. This option can be set globally or per org, repo and branch.

Important: If this option is not set and no prow jobs are defined tide will trust the GitHub combined status and will assume that all checks are required (except for it’s own tide status).

Example

tide:
  merge_method:
    kubeflow/community: squash

  target_url: https://prow.k8s.io/tide

  queries:
  - repos:
    - kubeflow/community
    - kubeflow/examples
    labels:
    - lgtm
    - approved
    missingLabels:
    - do-not-merge
    - do-not-merge/hold
    - do-not-merge/work-in-progress
    - needs-ok-to-test
    - needs-rebase

  context_options:
    # Use branch-protection options from this file to define required and optional contexts.
    # this is convenient if you are using branchprotector to configure branch protection rules
    # as tide will use the same rules as will be added by the branch protector
    from-branch-protection: true
    # Specify how to handle contexts that are detected on a PR but not explicitly listed in required-contexts,
    # optional-contexts, or required-if-present-contexts.  If true, they are treated as optional and do not
    # block a merge.  If false or not present, they are treated as required and will block a merge.
    skip-unknown-contexts: true
    orgs:
      org:
        required-contexts:
        - "check-required-for-all-repos"
        repos:
          repo:
            required-contexts:
             - "check-required-for-all-branches"
            branches:
              branch:
                from-branch-protection: false
                required-contexts:
                - "required_test"
                optional-contexts:
                - "optional_test"
                required-if-present-contexts:
                - "conditional_test"

Explanation: The component starts periodically querying all PRs in github.com/kubeflow/community and github.com/kubeflow/examples repositories that have lgtm and approved labels set and do not have do-not-merge, do-not-merge/hold, do-not-merge/work-in-progress, needs-ok-to-test and needs-rebase labels set. All PRs that conform to the criteria are processed and merged. The processing itself can include running jobs (e.g. tests) to verify the PRs are good to go. All commits in PRs from github.com/kubeflow/community repository are squashed before merging.

For a full list of properties of queries, please refer to https://github.com/kubernetes/test-infra/blob/27c9a7f2784088c2db5ff133e8a7a1e2eab9ab3f/prow/config/prow-config-documented.yaml#:~:text=meet%20merge%20requirements.-,queries%3A,-%2D%20author%3A%20%27%20%27.

Persistent Storage of Action History

Tide records a history of the actions it takes (namely triggering tests and merging). This history is stored in memory, but can be loaded from GCS and periodically flushed in order to persist across pod restarts. Persisting action history to GCS is strictly optional, but is nice to have if the Tide instance is restarted frequently or if users want to view older history.

Both the --history-uri and --gcs-credentials-file flags must be specified to Tide to persist history to GCS. The GCS credentials file should be a GCP service account key file for a service account that has permission to read and write the history GCS object. The history URI is the GCS object path at which the history data is stored. It should not be publicly readable if any repos are sensitive and must be a GCS URI like gs://bucket/path/to/object.

Example

Configuring Presubmit Jobs

Before a PR is merged, Tide ensures that all jobs configured as required in the presubmits part of the config.yaml file are passing against the latest base branch commit, rerunning the jobs if necessary. No job is required to be configured in which case it’s enough if a PR meets all GitHub search criteria.

Semantic of individual fields of the presubmits is described in ProwJobs.

2 - Maintainer's Guide to Tide

Best practices

1. Don’t let humans (or other bots) merge especially if tests have a long duration. Every merge invalidates currently running tests for that pool.
2. Try to limit the total number of queries that you configure. Individual queries can cover many repos and include many criteria without using additional API tokens, but separate queries each require additional API tokens.
3. Ensure that merge requirements configured in GitHub match the merge requirements configured for Tide. If the requirements differ, Tide may try to merge a PR that GitHub considers unmergeable.
4. If you are using the lgtm plugin and requiring the lgtm label for merge, don’t make queries exclude the needs-ok-to-test label. The lgtm plugin triggers one round of testing when applied to an untrusted PR and removes the lgtm label if the PR changes so it indicates to Tide that the current version of the PR is considered trusted and can be retested safely.
5. Do not enable the “Require branches to be up to date before merging” GitHub setting for repos managed by Tide. This requires all PRs to be rebased before merge so that PRs are always simple fast-forwards. This is a simplistic way to ensure that PRs are tested against the most recent base branch commit, but Tide already provides this guarantee through a more sophisticated mechanism that does not force PR authors to rebase their PR whenever another PR merges first. Enabling this GH setting may cause unexpected Tide behavior, provides absolutely no benefit over Tide’s natural behavior, and forces PR author’s to needlessly rebase their PRs. Don’t use it on Tide managed repos.

Expected behavior that might seem strange

1. Any merge to a pool kicks all other PRs in the pool back into Queued for retest. This is because Tide requires PRs to be tested against the most recent base branch commit in order to be merged. When a merge occurs, the base branch updates so any existing or in-progress tests can no longer be used to qualify PRs for merge. All remaining PRs in the pool must be retested.
2. Waiting to merge a successful PR because a batch is pending. This is because Tide prioritizes batches over individual PRs and the previous point tells us that merging the individual PR would invalidate the pending batch. In this case Tide will wait for the batch to complete and will merge the individual PR only if the batch fails. If the batch succeeds, the batch is merged.
3. If the merge requirements for a pool change it may be necessary to “poke” or “bump” PRs to trigger an update on the PRs so that Tide will resync the status context. Alternatively, Tide can be restarted to resync all statuses.
4. Tide may merge a PR without retesting if the existing test results are already against the latest base branch commit.
5. It is possible for tide status contexts on PRs to temporarily differ from the Tide dashboard or Tide’s behavior. This is because status contexts are updated asynchronously from the main Tide sync loop and have a separate rate limit and loop period.

Troubleshooting

1. If Prow’s PR dashboard indicates that a PR is ready to merge and it appears to meet all merge requirements, but the PR is being ignored by Tide, you may have encountered a rare bug with GitHub’s search indexing. TLDR: If this is the problem, then any update to the PR (e.g. adding a comment) will make the PR visible to Tide again after a short delay. The longer explanation is that when GitHub’s background jobs for search indexing PRs fail, the search index becomes corrupted and the search API will have some incorrect belief about the affected PR, e.g. that it is missing a required label or still has a forbidden one. This causes the search query Tide uses to identify the mergeable PRs to incorrectly omit the PR. Since the same search engine is used by both the API and GitHub’s front end, you can confirm that the affected PR is not included in the query for mergeable PRs by using the appropriate “GitHub search link” from the expandable “Merge Requirements” section on the Tide status page. You can actually determine which particular index is corrupted by incrementally tweaking the query to remove requirements until the PR is included. Any update to the PR causes GitHub to kick off a new search indexing job in the background. Once it completes, the corrupted index should be fixed and Tide will be able to see the PR again in query results, allowing Tide to resume processing the PR. It appears any update to the PR is sufficient to trigger reindexing so we typically just leave a comment. Slack thread about an example of this.

Other resources

• Configuring Tide

3 - PR Author's Guide to Tide

If you just want to figure out how to get your PR to merge this is the document for you!

Sources of Information

1. The tide status context at the bottom of your PR. The status either indicates that your PR is in the merge pool or explains why it is not in the merge pool. The ‘Details’ link will take you to either the Tide or PR dashboard.
2. The PR dashboard at “<deck-url>/pr” where <deck-url> is something like “https://prow.k8s.io”. This dashboard shows a card for each of your PRs. Each card shows the current test results for the PR and the difference between the PR state and the merge criteria. K8s PR dashboard
3. The Tide dashboard at “<deck-url>/tide”. This dashboard shows the state of every merge pool so that you can see what Tide is currently doing and what position your PR has in the retest queue. K8s Tide dashboard

Get your PR merged by asking these questions

“Is my PR in the merge pool?”

If the tide status at the bottom of your PR is successful (green) it is in the merge pool. If it is pending (yellow) it is not in the merge pool.

“Why is my PR not in the merge pool?”

First, if you just made a change to the PR, give Tide a minute or two to react. Tide syncs periodically (1m period default) so you shouldn’t expect to see immediate reactions.

To determine why your PR is not in the merge pool you have a couple options.

1. The tide status context at the bottom of your PR will describe at least one of the merge criteria that is not being met. The status has limited space for text so only a few failing criteria can typically be listed. To see all merge criteria that are not being met check out the PR dashboard.
2. The PR dashboard shows the difference between your PR’s state and the merge criteria so that you can easily see all criteria that are not being met and address them in any order or in parallel.

“My PR is in the merge pool, what now?”

Once your PR is in the merge pool it is queued for merge and will be automatically retested before merge if necessary. So typically your work is done! The one exception is if your PR fails a retest. This will cause the PR to be removed from the merge pool until it is fixed and is passing all the required tests again.

If you are eager for your PR to merge you can view all the PRs in the pool on the Tide dashboard to see where your PR is in the queue. Because we give older PRs (lower numbers) priority, it is possible for a PR’s position in the queue to increase.

Note: Batches of PRs are given priority over individual PRs so even if your PR is in the pool and has up-to-date tests it won’t merge while a batch is running because merging would update the base branch making the batch jobs stale before they complete. Similarly, whenever any other PR in the pool is merged, existing test results for your PR become stale and a retest becomes necessary before merge. However, your PR remains in the pool and will be automatically retested so this doesn’t require any action from you.