IMPORTANT: Read go/thirdparty first.
Third-party packages (e.g. a Red Hat
.src.rpm or a GNU
.tar.gz file) that
are compiled into a standalone packages or third-party binaries are checked into
vendor_src/. The package doesn't belong in
//third_party, because it doesn't
link to google3 code and no google3 code links to it. A corresponding directory
google_vendor_src_branch/ is used to house any Google-specific changes.
All patches or changes are made to
google_vendor_src_branch/. The code in
vendor_src/ is never touched beyond checking in new distributions from the
original authors (i.e. the "vendor"). Following this procedure makes it easier
to carry any Google-specific changes forward into new versions.
IMPORTANT: Adding a closed source, third-party binary to
something the go/ise team should audit.
NOTE: Historically, external code could be placed in
//piper/.../third_party. These places are deprecated and
new code is not allowed in them.
Adding the new package
IMPORTANT: New packages should probably go in //third_party, even if they don't use blaze. Please email third-party-removed for guidance before adding a new package to //piper/.../vendor_src.
Adding the new package is usually done in two CLs, with the first CL reviewed by emailremoved@ (see go/thirdparty/review#oscr) and including these parts:
- Add the code to
vendor_src/<package>/—but only the pristine unpacked tarball. Alternately, code can be imported from an external repo using copybara, although you will have to update METADATA by hand.
- If necessary, write a shell script that builds the package appropriately.
- Put the shell script and third-party metadata files (
google_vendor_src_branch/<package>/. Note that rules mentioning the
third_partytree can be treated as meaning the
google_vendor_src_branchtree, see go/thirdparty/documentation
The third-party metadata files need to be in this first CL in order to "bind"
vendor_src/ location to the
google_vendor_src_branch/ location in the
initial package creation CL. This makes it easier to track down from where the
vendor_src/ code was branched without having to traverse the changelog.
NOTE: If your pristine upstream package contains a OWNERS, or METADATA file you
will need to set
ALLOW_METADATA=<some reason> in your CL description.
Otherwise, there should not be a OWNERS, or METADATA in
The second CL, which does not need to be reviewed by emailremoved@, includes these parts:
- Integrate the pristine sources from
- If needed, re-open the files in
google_vendor_src_branch/<package>/for edit and make any changes.
Updating the package
Do the following to easily move to a new external version of the tool:
- unpack the new tarball there.
- integrate those changes to
Building the package
Building the package "appropriately" means:
- making sure any generated files (
.o, executables, etc..) go into a temporary directory, and not the Piper client, and
- building with the Crosstool compiler (see
the directions from the Crosstool team).
Do not use whatever compiler happens to be in
/usr/binon the computer you are using at the moment.
Blaze is not available in
vendor_src, so you can use any appropriate build
system. Many packages use standard Makefiles.
Adding and deleting the right files in
vendor_src/ can be a pain if you are
Comparison to //piper/third_party
Third-party packages live in multiple Google source code repositories,
//piper/.../google_vendor_src_branchcontaining Google modifications)
- Gerrit (go/gob)
//piper/.../vendor_src houses standalone tools based on third-party
code. Code housed in
//piper/.../third_party is intended to run on
Production servers, link with other google3 code, and use the google3 build
gconfig, Blaze, etc.),
Put simply, if
google3 is linking to the new package or the new package is
google3, the new package belongs in
//piper/third_party. If it's
a standalone package that is built with its own build system (e.g.,
; make ; make install,
rpmbuild, go/benz, go/autodeb), or a binary supplied
by a external vendor, then it belongs in
Historically, vendor_src contained things that did not use BUILD files to build or were otherwise independent from google3. The best practice as of February 2021 is that almost all new packages should be added to //third_party instead (even if they are independent from google3.) The tooling is significantly better to validate provenance and appropriate license metadata.
vendor_src and third_party have always shared the same restrictions on binaries and multiple versions, but they have not been enforced strictly in vendor_src. In 2019 we revamped the exception processes, so there is now a clear procedure for getting an exception granted if it is appropriate. Please do not hesitate to request one.
copybara import in to vendor_src
Optionally, go/copybara may be used here to perform the mechanics of extracting code from an external repo (e.g. git) and depositing it into a piper CL against vendor_src.
Unlike using copybara for
third_party, using transformations when importing
vendor_src are strongly discouraged. The code should be as pristine as
An example copy.bara.sky file for a simple git repo might look like:
load("//devtools/copybara/library/workflow", "exclude_paths") pkg = "somepkg" git_repo = "http://linkremoved/" core.workflow( name = "default", origin = git.origin( url = git_repo, ref = "master", submodules = "NO", ), origin_files = exclude_paths([ ".gitignore", ]), destination_files = glob( include = ["vendor_src/" + pkg + "/**"], ), destination = piper.destination(mode = "DRAFT_CL"), # If you require detailed revision history in piper, ITERATIVE may be used. mode = "SQUASH", # This workflow uses SQUASH, so authoring is not used authoring = authoring.pass_thru("Nobody <emailremoved@>"), transformations = [ metadata.squash_notes( prefix = "Import for " + pkg + ".\n\n", oldest_first = True, ), core.move("", "vendor_src/" + pkg), ], )