Go

This page documents rules for adding third-party Go code to //piper/third_party/golang.

IMPORTANT: Read go/thirdparty first. When creating a new directory, set *********************************** in the CL. For Go-specific questions, email emailremoved@.

What it's for

The third_party/golang directory is for code that is developed outside of Piper and imported into Piper using the //piper/third_party/golang/import.go tool. The purpose of this tree is to enforce common standards about how new versions of Go code are imported into Piper.

Open source code that is developed with Piper as the source of truth (and possibly exported) does not need to adhere to the requirements of the third_party/golang tree and should generally live elsewhere in third_party.

Code which is being temporarily developed in Piper with the intention of an eventual move to an external repository may be added to third_party/golang.

Go requirements

Go code checked in to third_party/golang must satisfy the following requirements:

Have the necessary required top-level files: OWNERS, BUILD, LICENSE, and METADATA.
Build using Blaze with no special options.
Follow a directory structure that is compatible with the open source Go toolchain. In particular, every directory may only have at most one Go package. It's a convention that the Go package itself be given a name identical to the directory that it lives under. This may cause the Go import path inside Google to stutter (e.g., "google3/third_party/golang/protobuf/v2/proto/proto") but is a consequence of differences between Blaze and the Go toolchain.
Have an entry under the Modules section of //piper/third_party/golang/CONFIGURATION, which is a mapping of directories under third_party/golang to the root Go import path for that module as respected by the open source Go toolchain. For example, the Go protobuf module has an external Go import path rooted at "google.golang.org/protobuf", but is stored internally under the protobuf/v2 directory.

The last two rules exist to enable a systematic and reliable way to resolve open source Go import paths to Google-specific Go import paths.

Directory naming

Each external module should be stored in Piper in a sub-directory of third_party/golang. The directory name should not start with go. If upstream is gofoo, just call it foo if foo is unique enough. If there are many foos, you can import "github.com/bar/go-foo" as barfoo. Groups of closely related packages are sometimes imported as bar/foo, though this is not required.

Importing code

Directories under third_party/golang must use the //piper/third_party/golang/import.go tool to manage importing of open source Go code into Piper. Entire modules should be imported even if only a small subset of the packages are needed. If the module contains more packages than necessary, a filter specified in the import configuration file can be used to import only the desired packages.

This section only covers functionally provided by the import.go tool. Users who use other tools (e.g., Copybara) to import or export code do so on at their own initiative and risk. The most technically challenging part of the import/export process is the mapping of internal and external Go import paths, which the import.go tool is specifically designed to handle. If you choose not to use import.go, please contact the emailremoved@ and let them know why the tool was insufficient.

First-time import

To import a Go module with import.go, first add an entry to the Modules section in //piper/third_party/golang/CONFIGURATION that maps the chosen Piper sub-directory to the open-source Go module path. For example:

[Modules]
foo: github.com/user/foo

Once an entry has been added, an import can be initiated with:

go run third_party/golang/import.go -dir=foo

It will automatically create the following files:

GOIMPORT/CONFIGURATION: This file specifies how the import ought to operate. Every new import copies the contents of //piper/third_party/golang/CONFIGURATION.default.
METADATA: An initial metadata file is created for the import. It is your responsibility to populate the "description" with something meaningful and any other attributes you deem necessary.

The following file may be present:

LICENSE: The default configuration file has a rule that attempts to rename common names for the license file as LICENSE. However, it does not catch all cases and it may be your responsibility to provide the correct renaming.

The following files will need manual modification:

BUILD: See /documentation/reference/thirdparty/documentation/#build.
OWNERS: See /documentation/reference/thirdparty/documentation/#owners.

By default, import.go pulls the latest version of the module. If a specific version is desired, it can be specified with the -version flag. For example:

go run third_party/golang/import.go -dir=foo -version=v1.0.0

The -version flag understands the special values "same" and "latest", which specifies that import.go should import at the current google3 version or the latest version, respectively. Specifying "same" is an error for the first import.

Updating imported code

Pull and merge the latest version of the module.

go run third_party/golang/import.go -dir=foo

Any google specific patches may result in merge conflicts. Review and fix any files that failed to patch. After fixing the conflicts, test your module and regenerate the MANIFEST and AUTOPATCHES files.

blaze {build,test} //third_party/golang/foo/...
go run third_party/golang/import.go -dir=foo -version=same

Getting a review

To declare that you know the rules on this page (or at least to get a faster review), make your CL description like:

Import `github.com/bar/foo` as `third_party/golang/foo`

This package will be used by:

[ ] Other third_party code
[ ] Packages in Google3

Running `import.go --dir=foo -check` reports success.

Checked that `blaze {build,test} //third_party/golang/foo/...` works.

Send your CL to *********************************** for review.

If you need to add multiple packages, put each new package into a separate CL. If you're adding a package and its dependencies, you may find it helpful to use go/fig to manage the stack of CLs.

Visibility

Depending on the intended scope of the imported code, we recommend handling visibility in one of two ways:

If the code is imported for third_party dependencies only:

This is the default, and is what import.go will generate:
```
visibility = ["//third_party/golang:__subpackages__"]
```
This will allow the packages to be imported by current and future third-party code, but the package cannot be used from the rest of google3.
If the code will be used by non-third_party google3 code:

We recommend creating a package group in the top-level BUILD file and referencing this in each package or rule that is part of the "public" API of the imported code.
```
package_group(
  name = "users",
  packages = [
    "//third_party/golang/...",
    "//path/to/your/project/...",
  ],
)
```
This keeps the visibility list centralized and provides an opportunity to explain who can use it, how to get added to the allowed users list, etc.

Reference

This section provides detailed documentation regarding the import.go tool.

Configuration files

The syntax of the configuration files is loosely based on INI files. It consists of a sequence of section headers (e.g., [Section], where each section consists of an ordered list of entries (e.g., name: val0 val1). The entry itself consists of an entry name, followed by zero or more values, and must be a single line. The value is the raw bytes of the value up until the next Unicode whitespace. If whitespace needs to be encoded in the value, it can be escaped using double-quoted or backtick-quoted strings. Comments are any bytes following an # character up until the next newline.

Example configuration file:

# This is a comment regarding Section.
[Section]
entry_name1: value1a "value1b has spaces"  # trailing comment
entry_name2: value2

Global configuration file

The //piper/third_party/golang/CONFIGURATION file is the global configuration file for the import.go tool. It has the following sections:

`Modules`

Modules is a mapping of sub-directories under third_party/golang to the root Go import path for an open-source Go module. Directory names and module import paths must be unique.

Example:

[Modules]
  foo: github.com/user/foo
  bar: github.com/user/bar

`PkgPathRewrites`

PkgPathRewrites is a filter that specifies whether to rewrite an open-source Go import path to some Google-specific Go import path. A rewrite rule is rarely needed at the global level except to rewrite import paths to point to Go packages that live outside third_party/golang. This filter is comprised of a set of replace rules:

replace specifies a Go import path to match on and an import path to replace it with. It will always be matched against the open-source Go import path, and the output must be an import path that operates correctly with Blaze. A replace rule cannot replace the output of another replace rule.

If the upstream package and the replaced package have different package names, then the original package name must be appended to the resulting path delimited with a semi-colon. This is commonly the case for rewrites to go_proto_library rules where the upstream package name may differ from the downstream package name, which typically has a _go_proto suffix.

Example:

[PkgPathRewrites]
  replace: golang.org/x/net/context => context
  replace: google.golang.org/protobuf/types/known/anypb => //piper/.../any_go_proto;anypb

Import configuration file

All modules imported by import.go must have an import configuration file located at GOIMPORT/CONFIGURATION within the sub-directory under third_party/golang that the module is imported into. The first time import.go imports a module, it copies the default //piper/third_party/golang/CONFIGURATION.default configuration into the module directory. Users are free to modify the import configuration file as they see fit.

It may have the following sections:

`ImportFiles`

ImportFiles specifies which files to include from the open-source module and which to exclude. By default all files are imported. This filter consists of a sequence of include and exclude rules:

include is followed by a regular expression pattern used to match files to include into Piper.
exclude is followed by a regular expression pattern used to match files to exclude from Piper.

The sequence of rules are evaluated in the order defined in the filter until the first match is found, resulting in either inclusion or exclusion. The regular expression pattern is matched against a path that is always relative to the module root with a leading '/' (e.g., /path/to/file.txt). The benefit of having a '/' prefix is that directories can be always be matched with /dirname/ and files always be matched with /filename$.

This filter is applied against paths as they appear in the open-source Go module.

Example:

[ImportFiles]
  # Exclude hidden files and directories (e.g., ".gitignore").
  exclude: /[.]

  # Include /api/services, but exclude everything else in /api.
  include: ^/api/services/
  exclude: ^/api/

  # Include everything else (implicit; shown for clarity).
  include: .*

`ImportRenames`

ImportRenames specifies whether to rename certain files or directories. The use of this feature is sometimes needed to work around Piper or Blaze limitations on allowed filenames. This feature implies that the relative path as seen in the open-source Go module may differ from the path as stored in Piper. When import.go performs the Go import path translation, it knows to consult this filter to derive the Google-specific Go import path. This filter consists of a sequence of sed commands:

sed is followed by a edit script in the same syntax as the popular sed tool.

The open-source relative path is passed through all sed commands in sequence where the output of the previous command is passed as the input to the next command. The final result is the path used. The lack of any commands implies that no renaming takes place.

Example:

[ImportRenames]
  # Rename dashes as underscores.
  sed: s:-:_:g

  # Strip parenthesis.
  sed: s:[()]::g

The above filter renames /(foo-bar)/baz as /foo_bar/baz. Note that both sed commands were used to produce the output path. Also note that this feature can be used to rename a directory by changing its name in the middle of a file path.

`RewriteFiles`

RewriteFiles specifies which Go source files to apply import path rewriting upon. The Glaze tool is run upon any directories containing files matched by this filter. By default, all Go source files are matched. It has syntax and semantics identical to the ImportFiles filter. The filter is matched against the potentially renamed file paths from the ImportRenames filter.

Example:

[RewriteFiles]
  # Exclude testdata and hidden files per behavior of the go tool itself.
  exclude: /testdata/
  exclude: /[._][^/]*$

  include: .*

`GlobalPkgPathRewrites`

GlobalPkgPathRewrites provides the module the ability to specify a Go import path rewrite for all code (both those within this module and also in other modules that may depend on this module). It consists of a set of replace rules similar to the PkgPathRewrites filter in the global configuration. These rewrites are only relevant for import paths that are rooted within the current module. This filter take precedence over the PkgPathRewrites filter in the global configuration.

WARNING: Carefully think about rules being added to GlobalPkgPathRewrite as they will affect all other imports that depend on this module. It is possible that LocalPkgPathRewrite better satisfies your use case.

Example:

[GlobalPkgPathRewrites]
  replace: github.com/golang/protobuf/proto => //piper/.../proto

Assuming this occurs within the module for "github.com/golang/protobuf", this rewrite alters all imports of "github.com/golang/protobuf/proto" to use "google3/net/proto2/go/proto" instead. However, if this filter were placed in the module for "github.com/user/foo", it would have no affect since "github.com/golang/protobuf" is outside the module path scope.

`LocalPkgPathRewrites`

LocalPkgPathRewrites provides the imported module the ability to specify Go import path rewrites for all code only within this module. It can be used to rewrite any import path (including standard library import paths). It consists of a set of replace rules similar to the PkgPathRewrites filter in the global configuration. This filter takes precedence over the GlobalPkgPathRewrites mentioned above.

`GoogleFiles`

GoogleFiles specifies which files are internal to Google. The import.go tool will delete any files not present in the upstream repository and not covered by this filter. It has syntax and semantics identical to the ImportFiles filter. The filter is matched against the potentially renamed file paths from the ImportRenames filter.

Example:

[GoogleFiles]
  # BUILD files are special to Blaze.
  include: /BUILD$

  # Treat files with a google_ prefix as internal.
  # This enables the injection of internal code in a separate file
  # without having to worry about patch conflicts.
  include: /google_[^/]+  # treat files with google_ prefix as internal

  # Exclude all other files as internal (implicit; shown for clarity).
  exclude: .*

Example import configuration:

# ImportFiles specifies which files to import from the upstream source.
[ImportFiles]
  # Exclude dot-prefixed files and directories (e.g., ".gitignore").
  exclude: /[.]
  # Include everything else (this is implicit behavior; shown for clarity).
  include: .*

# LocalPkgPathRewrites specifies a set of import path rewrites to apply locally.
[LocalPkgPathRewrites]
  # Replace all usages of glog with the Google-specific log package.
  replace: github.com/golang/glog => //piper/.../log

# ImportRenames specifies whether to rename any source files or directories.
[ImportRenames]
  # Rename common variants of license names to simply be LICENSE.
  sed: s:^/LICENSE([.](gpl|md|txt))?$:/LICENSE:I

# GoogleFiles specifies files added to the import to support use within google3.
[GoogleFiles]
  # Include files special to import.go.
  include: ^/GOIMPORT/
  # Include files special to Piper.
  include: ^/METADATA$  # see go/metadata
  include: ^/OWNERS$    # see go/owners
  include: /BUILD$      # see go/build
  include: /g3doc/      # see go/g3doc
  # Include files with "google3_" prefix (helps with local changes).
  include: /google3_[^/]+$
  # Exclude everything else (this is implicit behavior; shown for clarity).
  exclude: .*

Process pipeline

The import.go tool performs an import by running through the following sequence of operations.

Downloading the source

Go modules are downloaded from the Go module proxy (http://linkremoved/) rather than being cloned from a Git or Hg repository. Using the module proxy speeds up imports by an order of magnitude compared to cloning a repository (which needs to download the entire history of the module). Furthermore, it provides a stable backup for the original source should the upstream repository one day disappear (which is the case for some code already imported into Piper).

Using the module proxy is generally advantageous, but it does mean that trying to import the absolute latest version may be delayed by a latency on the order of minutes.

Applying manual patches

Manual patches may be applied to the downloaded source. Patches must be located in the GOIMPORT/PATCHES sub-directory and be named with a .patch suffix. They are applied in sorted filename order directly upon the downloaded source (prior to any file path renamings or Go import path rewrites).

The patch must be in the unified diff format with a single path prefix added to the file paths. This is the format typically provided by git diff or as downloaded from GitHub or Geritt.

Example patch:

diff --git a/integration_test.go bug ...est.go
index 9df20d0..f62e072 100644
--- a/integration_test.go
+++ bug ...est.go
@@ -28,8 +28,8 @@
 var (
    regenerate = flag.Bool("regenerate", false, "regenerate files")

-   protobufVersion = "3.7.0"
-   golangVersions  = []string{"1.9.7", "1.10.8", "1.11.5", "1.12"}
+   protobufVersion = "3.7.1"
+   golangVersions  = []string{"1.9.7", "1.10.8", "1.11.6", "1.12.1"}
    golangLatest    = golangVersions[len(golangVersions)-1]

    // purgeTimeout determines the maximum age of unused sub-directories.

Filtering and renaming files

To determine the set of files to import, the tool consults the ImportFiles filter to determine whether to include or exclude a file. If a file is to be kept, the ImportRenames filter is consulted next to determine whether the file (or the directory that it lives under) should be renamed.

Rewriting Go imports paths

All Go source files being imported (except for those excluded by the RewriteFiles filter) are reformatted to have Go import paths translated from the open-source paths to ones that are compatible with Blaze. For example, it is at this stage that an import path may be rewritten from "google.golang.org/protobuf/proto" to "google3/third_party/golang/protobuf/v2/proto/proto".

The translation goes through the following process:

It checks whether there is an applicable LocalPkgPathRewrites rule from the current import configuration and uses it if possible.
It checks whether there is an applicable GlobalPkgPathRewrites rule from the module that the import path is rooted within and uses it if possible.
It checks whether there is an applicable PkgPathRewrites rule in the global configuration and uses it if possible.
It attempts to resolve the Go import path by locating the Go module that the import path is rooted in and walking the file tree until it (hopefully) finds the specified Go package. For example, to resolve "google.golang.org/protobuf/encoding/protojson", we identify that it has a root prefix match with "google.golang.org/protobuf" for code located in Piper under the protobuf/v2 directory. Using the suffix, we identify the Go package located in protobuf/v2/encoding/protojson as protojson. Concatenating these pieces of information together results in an import path of "google3/third_party/golang/protobuf/v2/encoding/protojson/protojson".

Most import paths are resolved using the last mechanism. The need to specify a custom rewrite should be the exception, not the norm.

Applying automatic patches

A feature of import.go is the ability to perform auto-patching. It provides a generally easier user experience in making local changes to an import. It operates in two stages. For the sake of explanation, let v1.0 be the current version that is already imported into Piper and v1.1 be the version that is about to be imported into Piper.

The module is downloaded at the current version, v1.0, and all stages prior to this are performed on that temporary import (i.e., filtering, renaming, and import path rewriting). A patch is derived between that temporary import and the current import within Piper. This patch file contains the set of local changes that have been made on top of v1.0. If no local changes were made in prior imports, then this patch is empty.
The contents of that patch file are applied to the current working import of v1.1 (after filtering, renaming, and import path rewriting). In the common case, the patch of local changes relative to v1.0 will apply cleanly to v1.1. If not, the merge conflict will be reflecting the file with <<< and >>> markers.

This process is similar to how git rebase works.

Synchronizing to Piper

Up until this point, all changes are being made in a temporary working directory. At this stage, the tool compares the contents of the working import against the contents of the import directory in Piper:

Files that exist only in the working import, but not in Piper are copied to Piper as an added file.
Files that exist in both the working import and Piper, but differ in content will be modified (with the imported copy over-writing the Piper copy).
Files that exist only in Piper, but not the working import are deleted unless the GoogleFiles filter in the import configuration specifies that this is Google-internal file, in which case no changes are made.

Lastly, glaze is executed on all directories containing imported Go source files. This generates BUILD files with go_library and go_test rules derived from the imported Go source code.

Generated files

After importing a module, import.go generates several metadata files:

GOIMPORT/AUTOPATCHES is a directory containing diffs for all of the automatic patches that were applied. The diffs are a trimmed variant of the unified diff format where line numbers have been removed to reduce the probability of noise when unrelated changes occur within the same file. If there are no local changes to the module, then this directory is not generated.
GOIMPORT/MANIFEST is a list of all files within the import directory along with the expected checksum for those files. In the future, a pre-submit check will be added to third_party/golang to verify the contents of the MANIFEST file. The purpose of this check is to ensure that local changes are properly reflected in either manual patches or automatically-generated patches and surfaced for proper review.

If local changes have been made and the above files are stale, then they can be regenerated by running:

go run third_party/golang/import.go -dir=foo -version=same

To verify that the manifest is up-to-date, then run:

go run third_party/golang/import.go -dir=foo -check

Filtering external and internal files

If you need to exclude certain files or directories from being imported from the upstream source, specify so in the ImportFiles section of the import configuration.

You may want to consider the following files as being excluded from import:

The vendor directory (dependencies should be added explicitly to third_party instead of vendored)
BUILD files (if the project uses bazel)
GitHub OWNERS files
Dot-prefixed hidden files (e.g., .gitignore)

If you need to make sure the import does not delete any files that should be kept for Google-specific usages, then specify so in the GoogleFiles section of the import configuration.

You may want to consider the following files as Google-specific:

The GOIMPORT directory (for use by import.go)
OWNER, BUILD, LICENSE, or METADATA files
The /g3doc/ directory
Files with a google_ prefix (for use with Google-specific logic)

See existing configuration files for example usages.

Making local changes

There are two primary mechanisms for making local changes:

Manual patches: These patches are applied on the vanilla copy of the upstream source. The recommended use case for manual patches is the cherry-picking of upstream changes that have not yet landed in the release imported into Google. Keeping cherry-picks as discrete patches enables them to be easily added and removed.
Automatic patches: These patches are applied at the end of the process pipeline after all Go import path rewrites have occurred. Automatic patches provide an easy way for users to make local changes (i.e., they just directly modify the files in their CITC). However, they lack the ability to distinguish the purpose of one local changes from another. We recommend that users comment judiciously about the purpose of the change. For substantial changes, we recommend putting the main functionality in a Google-specific file (e.g., google_internal.go) and patch in the minimal number of hooks in the upstream code to call over to the internal logic. Doing so should reduce the probability of a merge conflict in future imports.

After making local changes, the auto-patches need to be regenerated:
```
go run third_party/golang/import.go -dir=foo -version=same
```

Changes to just import paths are generally more maintainable if performed through the use of LocalPkgPathRewrites in the import configuration file.

Auto-patching issues

If a patch won't apply due to merge conflicts, import.go will leave diff markers <<< and >>> in your files where the conflicts occur. You'll have to manually resolve the conflicts by editing the files.
To revert local changes that you've made to a file, you can run import.go with a sequence of -autopatch_include or -autopatch_exclude flags to enable or disable patching on specific files. The auto-patching filter follows the same semantics as the ImportFiles and GoogleFiles filter in the import configuration.