Vendor Source

Required reading

IMPORTANT: Read go/thirdparty first.

Using //piper/.../vendor_src and //piper/.../google_vendor_src_branch

See comparison to //piper/.../third_party.

Third-party packages (e.g. a Red Hat .src.rpm or a GNU .tar.gz file) that are compiled into a standalone packages or third-party binaries are checked into vendor_src/. The package doesn't belong in //third_party, because it doesn't link to google3 code and no google3 code links to it. A corresponding directory in google_vendor_src_branch/ is used to house any Google-specific changes.

All patches or changes are made to google_vendor_src_branch/. The code in vendor_src/ is never touched beyond checking in new distributions from the original authors (i.e. the "vendor"). Following this procedure makes it easier to carry any Google-specific changes forward into new versions.

IMPORTANT: Adding a closed source, third-party binary to vendor_src/ is something the go/ise team should audit.

NOTE: Historically, external code could be placed in //piper/.../src, //piper/.../third_party, and //piper/.../third_party. These places are deprecated and new code is not allowed in them.

Adding the new package

IMPORTANT: New packages should probably go in //third_party, even if they don't use blaze. Please email third-party-removed for guidance before adding a new package to //piper/.../vendor_src.

Adding the new package is usually done in two CLs, with the first CL reviewed by emailremoved@ (see go/thirdparty/review#oscr) and including these parts:

1. Add the code to vendor_src/<package>/—but only the pristine unpacked tarball. Alternately, code can be imported from an external repo using copybara, although you will have to update METADATA by hand.
2. If necessary, write a shell script that builds the package appropriately.
3. Put the shell script and third-party metadata files (OWNERS, LICENSE, and METADATA) in google_vendor_src_branch/<package>/. Note that rules mentioning the third_party tree can be treated as meaning the google_vendor_src_branch tree, see go/thirdparty/documentation

The third-party metadata files need to be in this first CL in order to "bind" the vendor_src/ location to the google_vendor_src_branch/ location in the initial package creation CL. This makes it easier to track down from where the vendor_src/ code was branched without having to traverse the changelog.

NOTE: If your pristine upstream package contains a OWNERS, or METADATA file you will need to set ALLOW_METADATA=<some reason> in your CL description. Otherwise, there should not be a OWNERS, or METADATA in vendor_src.

The second CL, which does not need to be reviewed by emailremoved@, includes these parts:

1. Integrate the pristine sources from vendor_src/<package>/ to google_vendor_src_branch/<package>/ (typically using g4 cp).
2. If needed, re-open the files in google_vendor_src_branch/<package>/ for edit and make any changes.

Updating the package

Do the following to easily move to a new external version of the tool:

1. g4-edit vendor_src
2. unpack the new tarball there.
3. submit vendor_src
4. integrate those changes to google_vendor_src_branch

Building the package

Building the package "appropriately" means:

• making sure any generated files (.o, executables, etc..) go into a temporary directory, and not the Piper client, and
• building with the Crosstool compiler (see the directions from the Crosstool team). Do not use whatever compiler happens to be in /usr/bin on the computer you are using at the moment.

Blaze is not available in vendor_src, so you can use any appropriate build system. Many packages use standard Makefiles.

Adding and deleting the right files in vendor_src/ can be a pain if you are not careful.

Comparison to //piper/third_party

Third-party packages live in multiple Google source code repositories,

• //piper/.../vendor_src (with corresponding //piper/.../google_vendor_src_branch containing Google modifications)
• //piper/.../third_party, and
• Gerrit (go/gob)

//piper/.../vendor_src houses standalone tools based on third-party code. Code housed in //piper/.../third_party is intended to run on Production servers, link with other google3 code, and use the google3 build system (BUILD files, gconfig, Blaze, etc.),

Put simply, if google3 is linking to the new package or the new package is linking to google3, the new package belongs in //piper/third_party. If it's a standalone package that is built with its own build system (e.g., ./configure ; make ; make install, rpmbuild, go/benz, go/autodeb), or a binary supplied by a external vendor, then it belongs in vendor_src/ and google_vendor_src_branch/.

Historically, vendor_src contained things that did not use BUILD files to build or were otherwise independent from google3. The best practice as of February 2021 is that almost all new packages should be added to //third_party instead (even if they are independent from google3.) The tooling is significantly better to validate provenance and appropriate license metadata.

vendor_src and third_party have always shared the same restrictions on binaries and multiple versions, but they have not been enforced strictly in vendor_src. In 2019 we revamped the exception processes, so there is now a clear procedure for getting an exception granted if it is appropriate. Please do not hesitate to request one.

copybara import in to vendor_src

Optionally, go/copybara may be used here to perform the mechanics of extracting code from an external repo (e.g. git) and depositing it into a piper CL against vendor_src.

Unlike using copybara for third_party, using transformations when importing into vendor_src are strongly discouraged. The code should be as pristine as possible.

An example copy.bara.sky file for a simple git repo might look like:

load("//devtools/copybara/library/workflow", "exclude_paths")

pkg = "somepkg"
git_repo = "http://linkremoved/"

core.workflow(
    name = "default",
    origin = git.origin(
        url = git_repo,
        ref = "master",
        submodules = "NO",
    ),
    origin_files = exclude_paths([
        ".gitignore",
    ]),
    destination_files = glob(
        include = ["vendor_src/" + pkg + "/**"],
    ),
    destination = piper.destination(mode = "DRAFT_CL"),
    # If you require detailed revision history in piper, ITERATIVE may be used.
    mode = "SQUASH",
    # This workflow uses SQUASH, so authoring is not used
    authoring = authoring.pass_thru("Nobody <emailremoved@>"),
    transformations = [
        metadata.squash_notes(
            prefix = "Import for " + pkg + ".\n\n",
            oldest_first = True,
        ),
        core.move("", "vendor_src/" + pkg),
    ],
)