homebrew simulation

Homebrew tap for osrf simulation software

osrf
37
65
Ruby
Build Status
brew test-bot GitHub action brew test-bot

homebrew-simulation

Homebrew tap for osrf simulation software

To use:

brew tap osrf/simulation
brew install gz-harmonic

Bottle status

Status Fortress Harmonic Ionic
gz-cmake gz-cmake2 gz-cmake3 gz-cmake4
gz-common gz-common4 gz-common5 gz-common6
gz-fuel-tools gz-fuel_tools7 gz-fuel_tools9 gz-fuel_tools10
gz-gui gz-gui6 gz-gui8 gz-gui9
gz-launch gz-launch5 gz-launch7 gz-launch8
gz-math gz-math6 gz-math7 gz-math8
gz-msgs gz-msgs8 gz-msgs10 gz-msgs11
gz-physics gz-physics5 gz-physics7 gz-physics8
gz-plugin gz-plugin1 gz-plugin2 gz-plugin3
gz-rendering gz-rendering6 gz-rendering8 gz-rendering9
gz-sensors gz-sensors6 gz-sensors8 gz-sensors9
gz-sim gz-sim6 gz-sim8 gz-sim9
gz-tools gz-tools1 gz-tools2 gz-tools2
gz-transport gz-transport11 gz-transport13 gz-transport14
gz-utils gz-utils1 gz-utils2 gz-utils3
sdformat sdformat12 sdformat14 sdformat15
collection gz-fortress gz-harmonic gz-ionic

To build bottles

The https://build.osrfoundation.org jenkins instance is used for building bottles with the following job
(configured in brew_release.dsl):

This jenkins job is triggered for pull requests when an administrator makes a comment
on the pull request that includes the phrase build bottle.
The job should appear in the GitHub build status interface for the latest commit:

GitHub build status interface screenshot

For example, #1157 was created after
running our release.py script
and this comment
triggered the bottle build, resulting in a successful bottle upload and a4793387.

Bottle builds are not triggered automatically for every pull request for several reasons:

  • Not all pull requests require a bottle to be rebuilt (such as
    #1007 that added this text to the README)
  • Successful bottle builds result in binary artifacts being immediately uploaded to our hosting provider
    so pull requests should be screened for malicious intent by administrators before triggering
    a bottle build.
    • This process differs from the approach taken by
      homebrew/homebrew-core
      whose bottles are hosted at GitHub Packages, which hosts files
      according to the hash of their contents rather than by filename.
      The homebrew-core CI jobs will build bottles for any incoming pull request,
      which will upload bottles directly to GitHub Packages, but the SHA hash associated with these bottles will not be
      easily available unless the pull request is merged by a homebrew maintainer.

Reasons to prefer building bottles in one pull request at a time

When releasing multiple packages in a short period of time, there may be
multiple pull requests open at the same time. While it may feel proactive to
apply the build bottle comment to multiple pull requests at once, please
keep the following issues in mind before doing so:

Issue #1990: bottle build may fail if target branch receives commits

As documented in #1990,
bottle builds may fail if a pull request is merged while another bottle build
is ongoing. Until this issue is resolved, if you trigger multiple builds at
once, merging any of them before all have completed may cause one of them to
fail. Valid strategies for avoiding build failures due to this issue include:

  • Build bottles for one pull request at a time and wait until that pull
    request is merged before starting another build. If merging the pull requests
    is not urgent and you receive GitHub notifications for pull requests to this
    repository, you can use the following procedure:

    • Check if any open pull requests have ongoing bottle builds. If so,
      come back later.
    • If there are no ongoing bottle builds, check if any pull requests have
      finished building bottles by looking for a commit with message
      “update bottle” and merge them.
    • Pick a remaining pull request, update its branch with the latest changes
      from the base branch and comment build bottle. The order is important;
      if you start a bottle building job before updating the branch with the
      latest changes, just let the build finish without updating the branch.
    • When you see a notifications for a commit with message “update bottle”
      in this pull request, start this process from the beginning.

  • Comment build bottle on any open pull requests and wait until all
    builds have completed before merging any pull requests. Please consider
    waiting to apply additional build bottle comments to new pull requests
    that are opened to avoid extending the wait time.

  • Cherry-pick the changes from multiple pull requests into a single pull
    request to allow the bottle updates to be built at once. This is also
    more efficient for formulae in the same dependency chain as it eliminates
    redundant checking in separate bottle building jobs. See
    #2949 and
    #2957 for examples
    of this approach.

To disable broken bottles

When a new major or minor version of a formula is merged to homebrew-core that is a dependency of formulae
in this tap, it may break our bottles, requiring a rebuild (see
#1728 or
#1708 for example).
As the osrf/simulation bottles will be broken immediately upon the merge in homebrew-core,
it can be useful to disable the broken bottles while waiting for new bottles to be rebuilt.
The --remove-bottle-block parameter to brew bump-revision can be used in this instance.
For example, the bottle removals from
8ca61f3d
in #1742 could be repeated with
the following commands:

brew bump-revision --remove-bottle-block ignition-msgs1
brew bump-revision --remove-bottle-block ignition-msgs5
brew bump-revision --remove-bottle-block ignition-msgs6
brew bump-revision --remove-bottle-block ignition-msgs7
brew bump-revision --remove-bottle-block ignition-msgs8

If all broken formulae properly list an explicit dependency on the breaking formula, the following
shell script loop can be used to remove all broken bottles. For example, the following
snippet was used to remove broken protobuf bottles in https://github.com/osrf/homebrew-simulation/issues/2314#issuecomment-1626396384:

cd `brew --repo osrf/simulation`/Formula
for f in $(grep -l '^ *bottle do' $(grep -rlI depend.*protobuf .) | sort)
do
  brew bump-revision --remove-bottle-block --message="broken bottle" $f
done

Troubleshooting

  • Does a new bottle need to be built for every homebrew pull request?

    • A new bottle is not needed for all pull requests.
      For example, updates to documentation or a formula’s test do block do not change
      the installed binary and thus don’t require a new bottle.
      Changing the tarball url, adding patches, or bumping the formula revision do
      require new bottles.
      If you aren’t sure, just ask in the pull request.

  • Do I need to wait for the brew test-bot / test-bot GitHub action to succeed for starting a bottle build?

    • No, you don’t need to wait. You can start the bottle build as soon as the pull request is opened.

  • When can I merge a pull request? Does CI need to be finished?

    • Yes, CI must be finished and successful. If a new bottle is needed, the
      generic-release-homebrew_triggered_bottle_builder job must be successful as well.

  • I commented build bottle, but it did not start a
    generic-release-homebrew_triggered_bottle_builder
    job.

  • I’m updating a Formula to build from a specific commit in a git repository.
    How do I get the sha256 for the tarball corresponding to that commit?

    • First, make sure that you have updated the url to use the commit hash that corresponds to the commit in the repository that you’d like to use.
      Also be sure to update the version, if it exists (in the example linked here, 20201028~c02cd0 is the part that needs to be modified: 20201028 is the date (year-month-day), and c02cd0 is the first 6 characters of the commit was used in the updated url).
    • Now, run the command wget <url>, where <url> is the updated url that was just mentioned.
      Once you have the tar file downloaded, run the command sha256sum <file>, replacing <file> with the file that was downloaded via wget.
      The sha256 will be printed to the console, which can then be used to update the Formula’s sha256.

  • I ran the release.py script multiple
    times for the same release and commented build bottle on the pull request, but the bottle building job failed,
    with console output containing the text Warning: Formula reports different SHA256:.

    • It’s possible that the tarball uploaded at the time the pull request was created was overwritten
      by a subsequent call to release.py (see ignition-tooling/release-tools#274).
      If so, update the sha256 field for the tarball (see #1156
      and 57fa5defcce for an example).

Jenkins implementation details

The generic-release-homebrew_triggered_bottle_builder
jenkins job currently builds bottles for macOS 10.15 catalina and 10.16 big_sur
using the following job configurations and the
homebrew_bottle_creation.bash
script:

If the bottle building job finishes without errors for each build configuration,
it will trigger a subsequent repository_uploader_packages
job that uploads the bottles to s3
and a generic-release-homebrew_pr_bottle_hash_updater
job that commits the changes in bottle sha256 values to the pull request branch
using this script.

Building bottles for newly supported macOS distributions

When we add support for a new version of macOS, we need to build bottles for that formula,
while ideally keeping the existing bottles. This can be done by using the --keep-old
parameter with brew test-bot and brew bottle.
Since ignition-tooling/release-tools#556,
bottle builds can be triggered for a specified version of macOS using --keep-old
by adding special tags to the build bottle comment in a homebrew-simulation pull request.
Use brew-bot-tag: along with build-for-new-distro-{distro} in the comment,
where {distro} is the version string used in homebrew bottle blocks
(such as catalina or big_sur). See this comment
in osrf/homebrew-simulation#1694
as an example that triggered a bottle build for big_sur only.
Note that the --keep-old flag only works if the pull request does not change the
formula version. Adding a comment to a formula (as in
osrf/homebrew-simulation#1694)
is sufficient.