Non-google3 Third Party Locations

IMPORTANT: This document is for code that lives outside of Piper. For code inside Piper see Put the Code in (the Right) //third_party.

TLDR: Put all non-Google code in a "third_party" directory in your repository. This policy applies to third-party code in ALL repositories outside of google3, including all projects on Gerrit, GitHub, etc.

What and why?

All non-Google code goes into the third_party/ directory of your project

Using third-party code saves time and is consistent with our values. To include any code with non-Google contributions to it in your project, the code must be placed in the designated third-party folder of your project. By default this should be a folder called third_party/ in the root directory of your project repository, but the exact name may vary by project.

Here is a non-exclusive list of code that must go into the designated third_party directory:

• Open source, free software, or commercially-licensed code needs to be in the designated third-party folder.
• Tools or libraries or protocols that are open source, free software, or commercially licensed, need to be in the designated third party folder.
• Derivative works of third-party code
• Library you wrote for your Ph.D. thesis
• Code you wrote at your last job
• Google modifications of third party code

But please do keep in mind that some software is only available under licenses that make it unsuitable for use at Google. Similarly, we generally don’t want to use such unsuitable code for open source projects either.

I just want one file, why can't I keep it with my other code?

Code gets copied, moved, and modified between projects, repositories, branches, etc. Single files placed outside third-party directories cause serious problems for other people trying to comply with open source licenses. People who use your project expect to be able to understand the licensing requirements. If most of your project seems like it's licensed under Apache-2.0, with just one non-compliant file hiding somewhere, the likelihood of your users falling out of compliance is very high.

Putting all code into the third_party/ directory of your project makes it easier to examine all non-Google code in a project-agnostic way

We respect other people's rights (both in letter and spirit). When we ship a new product or make a new open source release, we need to examine non-Google code to make sure we're complying with the authors' licenses. We also want outside developers to be able to easily comply with our licenses. When tools want to enforce Google-style, they ignore non-Google code. When there's a security warning about non-Google code, we want to scan all non-Google code to find problems and be able to alert people automatically (see go/vomit). All these are easier when non-Google code is in one place, instead of spread throughout the codebase. Hence, you place them in your project's designated third-party directory. And because third-party directories make life easier for the open source team, we have more time to help you.

Special Cases

I need to include a third-party image file in an Android project

In general, all images and assets should appear in the third_party directory of your project with appropriate LICENSE files in place. For Android projects, drawable resources must be located in the res/drawable directory and cannot have accompanying LICENSE files in that directory. Therefore, if you are including third-party image files in your Android project, you must include an ASSETS_LICENSE file in the root directory of your project with the concatenated license text of all the licenses for all third-party images located in the res/drawable directory of your project.

I need to include third-party source code in a Dart package.

In general, all source code should appear in the third_party/ directory of your project with appropriate LICENSE files in place. Dart packages have specific layout requirements. Dart sources must be located in the lib/ directory and non API sources should be located in the lib/src/ directory. Therefore, if you are including third-party Dart source code in your package, it must be located in the lib/src/third_party directory instead of third_party/. For mixed language projects, code that can be in third_party/ should be there.

Get the code

Get the code

When you find code you want to use, get it. This often means downloading from GitHub or elsewhere. Sometimes it can mean negotiating a license with another company and receiving the code another way. If you have any licensing concerns, contact emailremoved@.

Put the code in (the right) third party location

Find the right third party location

Most third-party code lives in a single repository in each Gerrit/other repo. If you can't find an appropriate third-party location, or one doesn't exist for your repository yet, create a folder in the root directory of your project called third_party/.

IMPORTANT: Android and Chromium have slightly different details.

Repository Name	Third party locations	Instructions
android	external/, platform/external/ 1	go/android3p
chromium	third_party/	go/chromium3p

Here is a simplified procedure for putting third-party code into the designated third-party directory of your project if one does not already exist:

1. Create a folder called third_party/ in the root directory of your repository. For Go repositories, you can use the name vendor/ for better compatibility with the Go build system.

2. Create a sub-directory for each non-google dependency.

3. In each sub-directory, make sure there is a file called LICENSE which contains the appropriate license text for the dependency. If one doesn't exist then create it. More details on this below.

4. Email emailremoved@ any time code is added, deleted, or modified within the designated third_party/ directory.

   NOTE: For some repositories (Android, etc), an auto-assigner has been set up. For those, you should use the auto-assignment instructions. If no instructions exist, send code reviews to emailremoved@.

5. The repository structure will look something like this:

   [Root Directory]
   |-- Google source code
   |-- ....
   `-- third_party
       `-- [external library A]
       |   |-- `LICENSE`
       |   |-- `METADATA`
       |   `-- ...
       `-- [external library B]
           |-- `LICENSE`
           |-- `METADATA`
           `-- ...
   

Following community conventions

Some tools and communities have well established tooling/patterns which serve the same purpose of delineating code ownership as outlined in this document, but does not rely on separating this code into a third_party directory.

Yarn

The yarn package manager has built in tooling for vendoring its usage within a project. This can be done using the yarn policies command as documented here. Yarn creates an entry in a project's .yarnrc file pointing to a single-file release file committed in the project in the .yarn/releases folder.

Separate out dependencies

Many open source projects include all of their dependencies. This is prone to versioning problems, and so we require that you separate out dependencies from code you import. These dependencies can be replaced by dependencies to other code in your third_party/ directory. There's a high chance that the depended-upon libraries are already there.

Why can there only be one version in a third-party location?

The transitive dependencies of most build systems mean that if there are two versions present, eventually someone will try to build a project that depends on both versions. Untangling this after the fact can be very time consuming. Worse, it can stop a project dead that was not involved in submitting either version.

Document the code and its context

By this point, your code should be working with Google integration. When you submit it, there needs to be enough context and background to help us with the requirements for third-party.

First, check in a pristine copy

The first commit should be the version of the code as it was downloaded. This allows us to track changes, and to revert back to the unedited version (e.g. we might change a file to work around a Google-specific quirk. If we remove the quirk, we want to remove the workaround). To make this easy, you may want to make a copy of this library in another client initially. The CL with the pristine copy should also contain LICENSE and METADATA files (except on Android, where we require NOTICE and MODULE_LICENSE_*). You do not have to include unused files, and you can move or rename files if necessary, but do not modify the contents of any files.

LICENSE must contain the full text of the license for the code.

Please note that, to be used at Google, third-party code must have some sort of license.

The license for this code must be in a file named LICENSE. If it was distributed like that, you're good. If not, you need to make LICENSE be a file containing the full text of the license. If there's another file in the distribution with the license in it, rename it to LICENSE (e.g., rename a LICENSE.txt or COPYING file to LICENSE). If the license is only available in the comments or at a URL, extract and copy the text of the license into LICENSE.

You may optionally document the generation of the LICENSE file in the local_modifications field of the METADATA file as follows:

local_modifications:
    "LICENSE file has been created for compliance purposes. "
    "Not included in original distribution."

If there are multiple licenses for the code, put the text of all the licenses into LICENSE along with separators and comments as to the applications. For additional information about how to deal with multiple licenses see the multiple-licenses section of the go/thirdpartylicenses.

The LICENSE file is used to automatically generate "About" or "Legal" screens. Please also wrap the LICENSE file to 80 characters and replace any non-ASCII characters with their ASCII equivalents so that such automated processes work well.

METADATA must list the information about this third-party code.

This allows people to quickly understand what the package is. A METADATA file looks like:

name: "Foo Framework"
description:
    "The Foo framework allows Bar frobbers to be registered. This avoids "
    "memory duplication as in //path/to/alternate/frobber:bar."

third_party {
  url {
    type: HOMEPAGE
    value: "http://linkremoved//"
  }
  url {
    type: GIT
    value: "https://github.com/yoyodyne/foo"
  }
  version: "X.Y.Z"
  last_upgrade_date { year: 2014 month: 10 day: 20 }
  license_type: NOTICE
  local_modifications: "Changed something important."
}

See go/thirdparty/metadata for explanations of the fields and examples.

The METADATA file is intended to capture all third-party metadata various teams use to track and analyze third-party packages. All code within third_party/ is required to have a METADATA file with a third_party field in the top-level directory of the package.

The local_modifications field is optional and is for informing other engineers who may use this package about modifications that may introduce behavioral differences between the checked-in version at Google and the pristine open source version. If you only make simple changes that do not affect the package's behavior, such as fixing compilation errors or renaming headers, you can just say "See git log" or omit the field entirely.

In google3, the license type of a package is declared in the BUILD file. Unless your project is using Bazel you don't need a BUILD file, so the license type should instead be declared in the METADATA file. If your license type ends up being by_exception_only, be sure to follow all the instructions at go/thirdpartylicenses.

Any other notes important to a Google user of this package must be listed in METADATA (see the proto definition for a full list of fields).

The non-HOMEPAGE URL should be to the versioned package you downloaded. Do not provide an unversioned URL or a URL to the project page.

For projects based on SVN, use the syntax:

url {
  type: SVN
  value: "http://<project>.<somehost>.com/"
}
version: "<revision>"

For GitHub projects, use the syntax:

url {
  type: GIT
  value: "https://github.com/<user>/<project>"
}
version: "<hash>"

NOTE: It's occasionally useful to include extra information along with the package that falls outside of third-party metadata—e.g., instructions on how to build the package, a list of local modifications made to the package, etc. While the metadata format is designed to be flexible enough to handle all use cases, it's not suitable for free-form text. For those teams that need this extra information, we suggest placing it in a README.google.md file. It has the advantage of keeping the data in a human readable format that can also be rendered as a webpage.

Get reviews

third-party-removed review

The initial submission of third-party code requires review from a member of emailremoved@. Subsequent changes that don't affect licensing don't require review by emailremoved@. You may modify the code as much as you want.

If you're using a Piper or Gerrit repository, you can have a third-party reviewer assigned automatically by setting up the go/gwsq auto-assigner. See the autoassigning page for details on how to set it up. Once set up, the auto-assigner will quickly assign reviewers to changes. Third-party reviewers generally respond within 1-2 business days.

If you have any questions, please contact emailremoved@.

Getting a security review

Third-party code is a hot spot for security vulnerabilities. When adding a new package that could potentially be used to process third-party / user content or otherwise carries any security risk, ping the emailremoved@. If it's a part of a more complex project, file a security review ticket for the entire thing, so that the security team can review the implications. This happens asynchronously, so it typically won't delay your launch. If security team has any concerns, you may be asked to add a security field to the METADATA file. When you update your code, be mindful of security-related mailing lists for the project and relevant CVE to update your package.

npm license checking

If you are using npm packages, you are responsible for making sure that all libraries you directly and indirectly use comply with the license policies.

Most npm packages have license metadata, but checking all dependencies by hand is painful.

NOTE: Many npm packages don't have proper license metadata in their package.json file. (Or they may not have it in the version you're using.) You'll need to check the source code of these by hand.

js-green-licenses

It is possible to parse the package.json file and extract the metadata, but it's a hairy mess due to the way the file is structured and the potential for multiple versions to be installed at once.

js-green-licenses will do this for you!

Yarn

Another option is to use the yarn package manager.

TIP: Yarn has other advantages like caching, a sane file format, a deterministic algorithm for installation, offline support, and more!

# In a directory with an existing npm install (i.e. package.json exists).

# 1. Install yarn
npm install yarn

# 2. Set up yarn: It will convert from package.json to yarn.lock
yarn install

# 3. List all packages with their license
yarn licenses list | less

# 4. Summarize licenses
yarn licenses list | grep License | sort -u | cut -d: -f2

Check the output of step #4 for any policy non-compliant licenses, and then you search the output of step #3 for those licenses to find the affected package(s). For packages with an UNKNOWN license, check the source code, as the metadata is sometimes missing information.

For policy non-compliant licensed packages, if you cannot find an alternative package, you could consider asking the author to re-license under a policy compliant license. (The majority of npm packages use the MIT license, so that is not a bad one to suggest.)