Haskell

This describes Haskell-specific guidance for checking code into //piper/third_party/haskell.

IMPORTANT: Read go/thirdparty first.

NOTE: The top level BUILD file is not required.

Ownership

Packages under //piper/third_party/haskell must have OWNERS files that comply with the general third-party guidelines. The majority of those packages are maintained by the Haskell language team, who take responsibility for tasks such as fixing breakages, approving cleanup CLs, upgrading to new versions or removing unused/obsolete packages.

We codify this arrangement with two states:

  • When an OWNERS file lists file://piper/.../OWNERS as its first line: it is maintained by the Haskell language team. Other FTEs may also be listed in that file as secondary maintainers.
  • When an OWNERS file lists two FTEs as the first two lines, they are responsible for maintaining that package. However, the Haskell language team may still proactively perform tasks such as fixes, upgrades and cleanups as needed.

In general, we recommend the former approach for new packages imported into google3. The latter is appropriate for more specialized projects that are not just mirrors of an external open-source package, or which are not using the automated process of go/cabal2build.

For questions or to report issues, contact removed link to google group or file a bug at go/haskell-bug.

Using third-party packages

Explicit packages

Packages that are not part of our GHC installation are located in subdirectories of //piper/third_party/haskell. They may be used as easily as any other Haskell library, by adding a dependency to a google3 BUILD rule in the normal way. Your BUILD file may look something like this:

haskell_binary(
    name = "Main",
    srcs = ["Main.hs"],
    deps = ["//third_party/haskell/hslogger"]
)

and your program will just do the obvious:

import System.Log  -- Will be coming from hslogger

Note that Blaze considers the target name //third_party/haskell/hslogger to be equivalent to //third_party/haskell/hslogger:hslogger. Other binaries and tests may be found in that same Blaze package (//third_party/haskell/hslogger:*).

Package names with dashes are converted to underscores; for example, the proto-lens library is provided by //third_party/haskell/proto_lens (equivalently: //third_party/haskell/proto_lens:proto_lens).

If your program needs a specific version of a third_party package, then it can depend on targets from that version explicitly, as follows:

haskell_binary(
    name = "Main",
    srcs = ["Main.hs"],
    deps = ["//third_party/haskell/hslogger/v1_0_7:hslogger"]
)

However, it is recommended that the non-versioned targets be used wherever possible, for reasons outlined in go/thirdparty.

Core packages

Some third-party Haskell packages are provided automatically by our GHC installation. They are listed in //piper/.../default_packages.bzl. "Default" packages (such as base) are available automatically to all build rules. Others (e.g., the ghc library which provides the GHC API) are available via explicit rules in //third_party/haskell/ghc:NAME, e.g., //third_party/haskell/ghc:ghc.

Installing and upgrading third-party packages

Automated installation

Most of the following procedure can be carried out automatically by the cabal2build tool. However, some manual intervention will still be needed before the result is ready to be checked in, so it's worth reading on.

The cabal2build tool is located at:

/path/to/.../cabal2build

You may also wish to add a shortcut:

alias cabal2build=/path/to/.../cabal2build

Read the manual like this:

cabal2build --help | less

Usual invocation looks like this:

cabal2build --fetch hslogger

As the final step of package import cabal2build should generate a corresponding versionless package (see below for motivation).

Dependencies

If your package X depends on another third-party package Y, install Y at the top level //third_party/haskell/Y instead of trying to put it as a subdirectory inside //third_party/haskell/X. A cabal2build invocation with --recursive does this automatically.

Each individual new package should be imported in a separate CL. One possible approach is to use DIFFBASE to send a sequence of CLs that depend on each other. This process may be slow, though, since some reviewers (especially third-party) may not want to approve CLs with a DIFFBASE.

Instead, we recommend the following, more parallel approach:

  1. First, import all of the packages with cabal2build --recursive --disabled. The --disabled flag will prevent cabal2build from generating a version-agnostic BUILD file, and it will also pass disabled = True to the cabal_haskell_package macro. That setting uses build tags to prevent Grok/Presubmit/etc. from trying to build the targets that the macro generates.

  2. Next, create separate (parallel) CLs, one for each package, and send them for review to third-party-*removed*. Since each build target is disabled, each CL can go through the third-party review process and be submitted independently, without being blocked by the CLs of its dependencies.

  3. Finally, send one last follow-up CL that enables all of the packages at once, by running cabal2build --wire-up on each of those packages. That will remove the disabled = True parameter and TODOs in the versioned BUILD file, and also add a version-agnostic BUILD file. (Note: if any packages require significant changes/patches to build, consider breaking up this CL into multiple steps.)

Note that by starting this process, you are taking responsibility for its completion: either by eventually enabling all of the imported packages, or by deleting any disabled packages which are no longer needed.

Internal patching

If patching is required to make the code compile or run properly, then it is preferred to submit the unchanged code first, and make the changes in a follow-up CL. Please add a note about this in the CL description.

If the code doesn't even compile, it should be submitted with the build rules commented out and a TODO note to make it compilable and reenable. Alternately, you may pass "disabled = True" to the cabal_haskell_package() macro.

This is most often needed for tests, which may make assumptions about test data file locations (you would need to set these up as data dependencies in the build rules, and use Google helpers to resolve the location at runtime). See //piper/.../FindFiles.hs.

Note that, by default, cabal_haskell_package generates the tests automatically. To disable that behavior (for example, when writing manual test rules), pass test = False to the cabal_haskell_package() macro.

Reviewers

Both emailremoved@ and emailremoved@ will automatically get added to your CL. We require a sort of "community policing" for language-specific third-party package domains, given that emailremoved@ cannot be fluent in all of these programming languages.

Upgrading an existing package

When upgrading an existing package to a newer version, if the license does not change, then you may submit the CL after approval from a Haskell-team member. You do not need to wait for approval from a member of third-party-removed as well. For the convenience of your reviewer(s), you may add a note like "Package upgrade only. No license changes."

If it is convenient, you may group together multiple related package upgrades into a single CL.

To upgrade to a new version of the package, you should create a parallel directory next to the existing one, then migrate all the existing users of the old version to the new one and remove the old one. If possible, remove the old version in the same CL that you add the new one. When you add a new version, you must commit to removing the old one or reverting the new within two weeks. See go/oneversion for an explanation.

If a two-step upgrade is necessary, you may create a directory for the new version of a package alongside the old one by passing the --keep flag to cabal2build:

cabal2build --keep --fetch hslogger

Users that depend upon the versionless package can all be migrated at once, just by migrating the versionless package, as follows (procedure identical to first-time installation):

cabal2build --fetch hslogger

This will:

  • fetch the latest version of hslogger
  • unpack it in the right directory
  • update the (versionless) BUILD file to point to the new package
  • g4/git/hg add or the files in the new directory
  • g4/git/hg rm the files in any previous versions.

You may keep old versions by passing the --keep flag. However, assuming everything builds/tests correctly against the new version, ideally you should remove the old version.

Adding a new package

Every new package must be added in its own CL. No other changes should be included in the CL.

Typical import from Cabal

Typical directory structure of a Cabal package imported from Hackage should look like this:

third_party/haskell/hslogger/
  BUILD
  OWNERS
  v1_2_10/BUILD
  v1_2_10/LICENSE
  v1_2_10/METADATA
  v1_2_10/src/BUILD
  v1_2_10/src/System/Log.hs
  ...

The version level v1_0_7/BUILD file exports the license file and describes how to build the package:

# Description: hslogger-1.2.10
#
# hslogger is a logging framework for Haskell, roughly similar to
# Python's logging module.
#
# hslogger lets each log message have a priority and source be associated
# with it.  The programmer can then define global handlers that route
# or filter messages based on the priority and source.  hslogger also
# has a syslog handler built in.
#
# Configured with Cabal flags:
#   buildtests: False
#   small_base: True

package(
    default_visibility = ["//third_party/haskell/hslogger:__pkg__"],
)

licenses(["notice"])

exports_files(["LICENSE"])

load(":package_description.bzl", "description")
load(
    "//tools/build_defs/haskell:cabal_package.bzl",
    "cabal_haskell_package",
)

cabal_haskell_package(description = description)

Please note the use of the cabal_haskell_package Skylark macro, which creates an expected haskell_library with an additional build_test pointing at it (see //piper/.../build_test.bzl?l=1 for details). This ensures that third-party Haskell libraries are included in a Presubmit project, preventing them from being broken accidentally.

The versionless BUILD file provides the same targets as the versioned google3 package:

haskell_library(
    name = "hslogger",
    deps = ["//third_party/haskell/hslogger/v1_2_10:hslogger"],
)

Usually Cabal packages include a Setup.hs file which has zero value when imported, hence please delete this file. The same goes for all other files which clearly have zero value (e.g. marked as obsolete or broken).

If a package includes a test suite, please attempt to use haskell_test target to wrap these tests into something Presubmit can test. This helps to avoid breakage of third-party Haskell packages by changing infrastructure.

Google3 package directories for Haskell package versions

Convert a Haskell package version suffix into a google3 package directory name by prepending v, and turning . or - into _ to make it valid. For example, hslogger-1.2.10 should be checked in a directory called v1_2_10 (under //piper/third_party/haskell/hslogger, of course).

Multiple versions

Haskell allows multiple versions of the same package to coexist in the same binary as long as they are in different packages. Hopefully we will not need to have that, but it is possible.

Tests

If the original package has any tests, they should be exposed to blaze in the BUILD file.

Versionless third-party Haskell packages: motivation

When upgrading a Haskell package that is installed in google3 under //third_party/haskell/${package}, it's quite hard to migrate google3 packages that depend on it. That's because the new version is a new google3 package (//third_party/haskell/${package}/v${YYY} instead of //third_party/haskell/${package}/v${XXX}) so all the dependencies have to be updated.

This is eased by introducing trampoline targets like //third_party/haskell/${package}:${package}. Such a target provides a single point at which to switch versions. The versionless packages are a fundamental assumption built into go/cabal2build and go/haskell-rules.