Third Party METADATA

METADATA file

The METADATA file is intended to capture all third-party metadata various teams use to track and analyze third-party packages.

All directories within //third_party are required to have a METADATA file with a third_party field, unless they are below the root directory of a package. This rule does not apply to the following directories, which have slightly different requirements:

• //third_party/java{_src}
• //third_party/kotlin{_src}

It's occasionally useful to include extra information that falls outside of third-party metadata—e.g., instructions on how to build the package, a list of local modifications made to the package, etc. While the metadata format is designed to be flexible enough to handle all use cases, it's not suitable for free-form text.

For those teams that need extra information, you can place it in a Markdown file (README.md, README.google.md, etc.), in a g3doc subdirectory, etc., which best aligns with your use case.

The third_party struct

TIP: Place the third_party field near the top of the METADATA file. That way people can quickly view third-party metadata.

The third-party metadata resides in a third_party field in the METADATA file. Third-party metadata can describe several types of directories, which require different sets of fields.

type

The type field documents what type of third_party record this directory represents. It may be one of several values:

• PACKAGE (default)—This directory represents a third-party package, complete with a LICENSE file, BUILD file, etc.

• VERSIONS—This directory contains several PACKAGE directories representing different versions of the same third-party project. See go/oneversion#exceptions.

• GROUP—This directory is designed to organize multiple third-party PACKAGE directories. See go/thirdparty/where#subdirectories.

• GOOGLE_INTERNAL—This directory contains no third party packages or code, only internal docs, tools, and other Google-authored files for internal use.

Each of these require the top-level name and description fields be set. For type: GROUP and type: GOOGLE_INTERNAL, the description field should document the reason the directory is needed.

type: PACKAGE

The normal required fields are: version, and last_upgrade_date. The url field identifies where it came from: ARCHIVE, GIT, SVN, HG, PIPER, OTHER, or LOCAL_SOURCE.

NOTE: A PIPER URL doesn't need the version and last_upgrade_date fields if the value is referring to itself. A LOCAL_SOURCE URL doesn't need the last_upgrade_date field.

Information about security and multiple versions is also encoded in the METADATA file. (See the third-party metadata.proto file below for information on how to fill out all fields in third_party.)

Example:

name: "jQuery"
description:
    "jQuery is a fast, small, and feature-rich JavaScript library. It makes "
    "things like HTML document traversal and manipulation, event handling, "
    "animation, and Ajax much simpler with an easy-to-use API that works "
    "across a multitude of browsers."

third_party: {
  type: PACKAGE  # `type` defaults to `PACKAGE`, so typically this line is omitted
  url {
    type: HOMEPAGE
    value: "https://jquery.com/"
  }
  url {
    type: GIT
    value: "https://github.com/jquery/jquery"
  }
  version: "2.2.2"
  last_upgrade_date {
    year: 2014
    month: 10
    day: 20
  }
}

type: VERSIONS

If you are putting source code into a version named directory (i.e //third_party/package/v1.0) then the METADATA in //third_party/package must contain third_party { type: VERSIONS }.

Each versioned directory (i.e. v1.0) must have its own METADATA file with a proper third_party block.

Multiple Versions

IMPORTANT: Use of type: VERSIONS does not grant permission to have multiple versions.

If you have been cleared to include multiple versions of the same package in third_party/, you will also need the multiple_versions subfield to document the exception.

Within the multiple_versions subfield, there should be an approval that links to a relevant exception bug and a type which may either be PERMANENT or TEMPORARY. If it is TEMPORARY, there should also be an expiration to document the intended duration of the extra version(s) of the package.

Example:

name: "jQuery"
description:
    "jQuery is a fast, small, and feature-rich JavaScript library. It makes "
    "things like HTML document traversal and manipulation, event handling, "
    "animation, and Ajax much simpler with an easy-to-use API that works "
    "across a multitude of browsers."

third_party: {
  type: VERSIONS
  multiple_versions {
    type: TEMPORARY
    expiration {
      year: 2017
      month: 1
      day: 1
    }
    approval: "b/12345"
  }
}

type: GROUP

Typically, new packages either belong at the root of third_party/ or in a language-specific subdirectory. However, new subdirectories are occasionally desirable. These subdirectories need to be documented with a METADATA file. The name field should be the path relative to third_party/ and the description should explain why a subdirectory is needed.

Example:

name: "acme/fonts"
description:
    "Fonts produced by ACME.  These packages are grouped together in order to"
    "simplify our tooling."

third_party: {
  type: GROUP
  group: {
    approval: "b/1234567"
  }
}

type: GOOGLE_INTERNAL

Occasionally, teams will organize several packages in a single type: GROUP directory, but want an additional subdirectory to store files for internal use, such as docs or tools. In these instances, the subdirectory should be marked with type: GOOGLE_INTERNAL.

NOTE: This type should not be used for published code, even if it has not yet accepted (or will not ever accept) outside contributions. Instead, use PACKAGE

Example:

name: "special_architecture_packages/g3doc"
description:
    "Documentation for our use of third party packages related to "
    "special_architecture."

third_party: {
  type: GOOGLE_INTERNAL
}

last_upgrade_date

This field documents the last time the version in third_party was upgraded from an upstream source. When populating this field, use today's date, not the date that upstream published the version you're checking in.

The security team uses the last_upgrade_date field to determine when the last sync to upstream was performed. With that information, they can determine if a security fix was applied to the source. Therefore, the last_upgrade_date should not be updated if the change is a local modification. Change this field only when upgrading the code from upstream.

url

It's important that we can identify the precise URL the code was retrieved from so that, among other things, the security team can correlate it with any known security vulnerabilities. The URL field has a lot of flexibility in order to accommodate the many different ways third-party packages are retrieved.

This is a list of examples of each type of URL to clarify some of their uses:

• HOMEPAGE—The homepage for the package. This is optional but encouraged to help disambiguate similarly named packages or to get more information about the package. This is especially helpful when no other URLs provide human readable resources (such as git:// or sso:// URLs).

  url {
    type: HOMEPAGE
    value: "https://jquery.com/"
  }
  

• ARCHIVE—The URL containing the source code for the specific version of the package.

  url {
    type: ARCHIVE
    value: "http://linkremoved/"
  }
  

• GIT—The upstream git repository. This requires that the "version" field value specifies a specific git tag or commit hash.

  url {
    type: GIT
    value: "git://git.kernel.org/pub/scm/libs/netlink/libnl.git"
  }
  version: "02b871d7bc4c364e6e8b5d9c777eac49979e94e5"
  

  url {
    type: GIT
    value: "https://github.com/tensorflow/tensorflow"
  }
  version: "v0.9.0"
  

  Similarly for SVN and HG URLs.

• PIPER—Used when a package is being copied into third_party from elsewhere in piper but is actively being developed outside of third_party:

  url {
    type: PIPER
    value: "http://linkremoved//"
  }
  version: "cl/12345"
  last_upgrade_date: { year: 2001 month: 01 day: 02 }
  

  When a package is being newly developed in or moved to third_party, the PIPER URL should reference the package itself.

  url {
    type: PIPER
    value: "http://linkremoved/"
  }
  

  NOTE: The version and last_upgrade_date fields are only necessary if the third_party package is not the source of truth in PIPER.

  IMPORTANT: If a directory with a PIPER URL is released externally, the url type should be updated to type GIT and point to the external location.

• OTHER—A URL that does not fit any other type. This may also indicate that the source code was received via email or some other out-of-band way. This is most commonly used with commercial software received directly from the vendor. In the case of email, the URL value can be used to provide additional information about how it was received.

  url {
    type: OTHER
    value: "Delivered by Hedwig via Owl Postal Service"
  }
  

• LOCAL_SOURCE—The location of the local copy of the package source code.

  Typically, the metadata files describing a package reside in the same directory as the source code for the package. In a few rare cases where they are separate, the LOCAL_SOURCE URL identifies where to find the source code. This only describes where to find the local copy of the source; there should always be an additional URL describing where the package was retrieved from.

  Example cases of when to use LOCAL_SOURCE:

  • java packages that build from source (go/thirdpartyjava#format3) where source is in //third_party/java_src and metadata files are in //third_party/java
  • packages that follow go/vendorsrc
  • packages that follow go/android3p to sync an entire remote git repo to git-on-borg

  url {
    type: LOCAL_SOURCE
    value: "http://linkremoved/"
  }
  

  Example Git-on-Borg package that imports the full project history:

  url {
    type: LOCAL_SOURCE
    value: "https://android.googlesource.com/platform/external/apache-http"
  }
  

  NOTE: The last_upgrade_date field is not necessary for LOCAL_SOURCE URLs.

The updatemd tool

To help keep the METADATA file up-to-date, you can use the updatemd program to update commonly changed fields. For example:

$ /path/to/.../updatemd --help
Usage: updatemd [options] path

  -archive_url=""     : update the third-party "archive_url" field
  -print=""           : print the value of the field
  -version=""         : update the third-party "version" field