Java

This describes Java-specific guidance for checking code into //piper/third_party/java.

IMPORTANT: Read go/thirdparty first. For //piper/third_party/java specific questions, email emailremoved@.

NOTE: The //piper/third_party/java rules are only required for google3 code. If you are submitting code into a different depot, you should consider having //piper/third_party/java mirrored into your depot (see go/piper-task-bug), which gives you the benefit of all the maintenance work that goes into the primary depot, but this is not required.

Comments from experience

Most open source code is very good about being backwards compatible. Upgrading to new versions is often no more than giving teams a chance to verify that running against the new version still works.

IMPORTANT: If something doesn't work after an upgrade, rollback and then figure it out. The cost of broken builds across the company is extremely high.

Individual teams may say, "We don't want to change".

  • If a team can justify why the upgrade can't be done or needs to be delayed, listen.
  • If they can't justify it, tough. Going through these upgrades is part of the price of using the google3 build system. For most upgrades, two weeks should be plenty of time for a team to find time to test the change and possibly make minor changes to their code.

If you are not sure what to do at any point, email emailremoved@

Adding a new library

Check your preconditions

  • Verify the library is not already there under a slightly different name. The current naming is unfortunately often not consistent.
  • Verify there isn't already another similar library present. For example, there are many json parsers or logging frameworks. Please use one of the ones already present.
  • Ensure all libraries your new package depends on are present at an acceptable version. If your new package depends on other third party packages, do not submit those additional jars within your new package. This will lead to broken builds (see go/oneversion). Instead:
    • Depend on existing rules //piper/third_party/java for that supporting package.
    • If the exact version needed is not present, you must either use the existing version, or upgrade all of google to use the new version you require.
    • If the supporting package does not exist in //piper/third_party/java, add it in a separate changelist as a precondition of your current changelist.

Take the third_party/java owners oath.

# TODO: Hold up your right hand and repeat this.
#
# WE THE OWNERS HEREBY PROMISE to ensure that there will only be one version
# of this project in google3 at a time.
#
# If at any time we g4 approve a new version, we promise to work with the
# submitter to ensure that:
#
#  1) All of Google can move to the new version in one CL.
#
#  2) If a new version will break other teams ...
#     a) We will not submit until we find a way to do so without breaking them.
#     b) If we can update all but 1-2 teams without breaking those other teams,
#        and we really need this version now, we can submit providing we commit
#        to being back on one version Google wide within TWO WEEKS.
#
#  3) The old version will be deleted within two weeks of submitting
#     the new version.
#
#  4) Any dependencies of this package will be separate third_party/java
#     entries.  (No /lib directories or equivalent).
#
#  5) We will always depend on the unversioned rule for dependencies.
#     (e.g. //third_party/java/foo, not //third_party/java/foo/v1_3)
#
# IF THE UPGRADE SUBMITTER DOES NOT FOLLOW THROUGH, WE PROMISE THAT WE
# WILL EITHER FINISH FOR THEM OR IMMEDIATELY DELETE THE NEW VERSION.

Create the new package

There are three common formats that you will find for Java projects in //third_party.

NOTE: Starting in late 2019, third party Java usage at Google has standardized on moving to source based builds by default, previously known as "format 3." This is now described as the "preferred format" below.

In each case:

  • Copy the appropriate templates, and fix the TODO's in each copied file. Please remember to remove the TODO lines after you address them.
  • In some cases there is a README.md file to provide additional requirements and guidance.
  • The initial changelist should contain a single new third-party package and nothing else (treating any //piper/third_party/java package and the corresponding //piper/third_party/java_src package as a "single package" in such cases). This submits the pristine sources and the go/thirdparty "metadata" together in a single, atomic changelist.
  • The go/thirdparty METADATA file belongs in the java_src tree. (Legacy libraries may have it in the java tree.)

(All of the commands below assume you start in the google3 directory.)

Preferred format: Source based packages

MY_NEW_PACKAGE=my_new_package
mkdir -p third_party/java_src/$MY_NEW_PACKAGE third_party/java/$MY_NEW_PACKAGE
cp -R /path/to/.../* third_party/java_src/$MY_NEW_PACKAGE/
cp -R /path/to/.../* third_party/java/$MY_NEW_PACKAGE/
chmod 640 $(find third_party/java/$MY_NEW_PACKAGE third_party/java_src/$MY_NEW_PACKAGE -type f) # ensure files found by g4 nothave
chmod +w third_party/java/$MY_NEW_PACKAGE third_party/java_src/$MY_NEW_PACKAGE

Although the source code is checked in to the java_src package and users compile it at build time, a //piper/third_party/java package is still required, as users only ever depend on packages in //piper/third_party/java.

To help facilitate regular upgrades as part of the maintenance requirements for packages in //third_party, you may find it useful to automate imports from upstream sources of truth where possible using copybara. The sample package directories include an empty copy.bara.sky that you can modify, but copybara can also synthesize one based on the contents of your METADATA file:

copybara info third_party/java_src/$MY_NEW_PACKAGE/METADATA --metadata-generate-common-libraries --metadata-disable-url-translation --verbose

You can apply local changes by defining Copybara transformations. and can run the import with the following command:

g4 change # to create a change
copybara --init-history --destination-cl=$(/path/to/.../vcstool pending-change-number) third_party/java_src/$MY_NEW_PACKAGE/copy.bara.sky

To make regular upgrades even easier, if you register your configuration with Copybara-as-a-Service, upgrade changelists will be automatically generated for review when the conditions you specify are met.

NOTE: Allowing local modifications for packages primarily developed elsewhere may cause pain long-term when upgrading to a new version. It might be worth insisting that issues are fixed upstream and the fixes imported using copybara. See bug ...nt2 for context.

Reference documentation for the other, deprecated formats remains below for convenience. Use of these formats requires an exception, which is verified by the BannedExtension check of go/compliancelint. A list of active exceptions can be found at go/empty-jars-exceptions.

By Exception Only / Deprecated Format 1: Jar files

IMPORTANT: This format is considered deprecated and should be used only when source code is not available. To submit a new package, or upgrade an existing package where only a .jar is available, fill out the exception template at go/third-party-removed. The component is managed by blunderbuss and will be reviewed by ISE-TPS with a target SLO of under 4 business days for response.

MY_NEW_PACKAGE=my_new_package
cp -R /path/to/.../* third_party/java/$MY_NEW_PACKAGE
chmod 640 $(find third_party/java/$MY_NEW_PACKAGE -type f) # ensure files found by g4 nothave
chmod +w third_party/java/$MY_NEW_PACKAGE

In this format, you never submit the expanded source. You build locally and only submit the updated jars. Be sure to include a -src.jar or -sources.jar if source is available, and read the general instructions.

New .jar-only packages should have restricted visibility and an appropriate Category field set in the METADATA file, determined as part of the exception consultation with ISE-TPS.

By Exception Only / Deprecated Format 2: Jar files plus source

This format requires extra work over Deprecated Format #1 with little to no benefit, so most users that want to commit the source should use the preferred source-only format. Rarely, the overhead of compiling the package may be large or the tooling cumbersome, in which case this format could be the best fit. To submit a new package in this format, fill out the exception template at go/third-party-removed. The component is managed by blunderbuss and will be reviewed by ISE-TPS with a target SLO of under 4 business days for response.

MY_NEW_PACKAGE=my_new_package
cp -R /path/to/.../SAMPLE_JAVA_SRC_PACKAGE third_party/java_src/$MY_NEW_PACKAGE
cp -R /path/to/.../SAMPLE_NEW_PACKAGE third_party/java/$MY_NEW_PACKAGE
chmod 640 $(find third_party/java/$MY_NEW_PACKAGE third_party/java_src/$MY_NEW_PACKAGE -type f)  # ensure files found by g4 nothave
chmod +w third_party/java/$MY_NEW_PACKAGE third_party/java_src/$MY_NEW_PACKAGE

In this format, you submit both the source code (in the java_src package) and compiled jars (in the java package). Be sure to include a -src.jar or -sources.jar in the java package as well.

Warnings and errors in //third_party code

Google's Java compiler adds additional bug checkers using go/errorprone, so third-party code may not compile with Blaze. error-prone team tries to ensure that our additional checkers detect high-value bugs, so we strongly encourage you to fix any bugs it points out and either maintain a local patch or push the changes upstream. However, we understand that in some cases this may not be possible, and you will need to disable the checks. There are two ways to do this:

  1. You can disable a specific check by passing an additional java compiler option (go/be#java_library.javacopts), "-Xep:checkname:OFF", to the BUILD rule. The checkname is given in brackets in the error message. For example, given the error message:

    [LongLiteralLowerCaseSuffix] Prefer 'L' to 'l' for the suffix to long literals

    you would pass the compiler option "-Xep:LongLiteralLowerCaseSuffix:OFF" to disable it. You can disable multiple checks by passing a separate flag for each one. Note that only certain checks are allowed to be disabled; you will get an error message if you try to disable a non-disableable check.

  2. You can disable all suppressible error-prone checks by passing the java compiler option (go/be#java_library.javacopts) "-XepAllErrorsAsWarnings".

Local modifications

The initial changelist should not include any local modifications.

When making local changes to the third-party source code, a "pristine" copy of the code, together with the necessary Google-specific files, must be submitted in the initial changelist . Local modifications must then be made in a follow-up changelist. This subsequent changelist can be reviewed by the custodial owners of the new package (those in the newly created OWNERS file), unless there are license or security considerations associated with the modifications that require review by others.

If you are using jar-only format, where we never check in the expanded source, please do submit an unmodified x.jar and x-src.jar/x-sources.jar before submitting a modified x.jar and x-src.jar/x-sources.jar and patch file. (Note that x-src.jar/x-sources.jar must always have the same directory structure as x.jar; this is not considered a "local modification".)

When you are ready to submit

blaze build third_party/java/$MY_NEW_PACKAGE:all to look for typos.

Send the CL for review:

  • add to R=
    • another new owner
    • and emailremoved@ (for legal/license issues),
    • and emailremoved@ (for everything else).

Both groups must LGTM the CL. The second to do so will approve.

Resolving conflicts between multiple versions

Cleaning up conflicts, including existing multiple versions, is very similar to doing an upgrade.

If the conflict is caused by two versions of the same package in third party java:

  • Please upgrade everyone to the more recent version and remove the older version. Instructions—start at upgrade CL#2 below.

If the conflict is caused by classes embedded in another jar:

  • Remove all embedded dependencies from the jar and depend on the official versions checked into third_party/java instead. For example, if the selenium jar also contained a version of junit, you would remove the junit classes from the selenium jar and add //piper/third_party/java/junit to the //piper/third_party/java/selenium:selenium rules dependencies. Script /path/to/.../remove_files_from_jar does the heavy lifting for this.
  • If the version of the conflicting classes you need is not in third_party/java, and the version you need is older than the version submitted in third_party/java:
    • Try using the newer version. Often it just works
    • If the code requiring the older third party library is Google code, please upgrade it to work with the newer version already in //piper/third_party/java.
    • If the dependent code is another third party library, please find a newer version that will work and follow the upgrade procedures below to move everyone to the newer version.
  • If the version of the conflicting classes you need is not in //piper/third_party/java, and the version you need is newer than the version already in //piper/third_party/java:
    • Try using the older version. Sometimes you get lucky and it just works.
    • If that doesn't work, please follow the upgrade procedures at to add the newer version and move everyone to it.

Upgrading an existing library (Note: There is no "add new version.")

The ground rules

When you add a new version, you are committing to upgrading all of Google to use the new version in one CL. You may not use the new version in your project beyond compatibility testing before this. The old version should be deleted as soon as it looks like the conversion CL will not need to be rolled back. The multiple version layout is only to ease the testing of the new version prior to converting all of Google. You may not just introduce a new version. See go/oneversion for an explanation why.

Man, that sounds harsh, doesn't it? Actually, it is a whole lot easier than you think. In practice, we find that if you follow the procedure outlined below, most upgrades take 4-8 hours of your time, spread across several days or weeks; the (rare) really, really ugly ones requiring chains of upgrades can take 2 weeks of time over 4-6 weeks. It turns out that having one person pay this cost and coordinate for everyone is a bargain to the company. Do you know of any other company where you could do this in less than a day of your time? How much time would each project need to spend to do this on their own? How much would be lost in version conflicts in the meantime?

Exceptions

The sole exceptions to this rule is JDBC drivers under //piper/third_party/java/jdbc. For JDBC, moving databases is such a hard task that the decision was made several years ago that all database dependencies can only be specified on the binary build target. You may not specify them as a dependency for a library build target. This avoids almost all of the version skew issues. (If your binary has a data dependency on another binary, it can still bite you.)

The three CLs

If the third_party library you are upgrading has very few users, you may do all of these in a single CL. Submit this CL for review to emailremoved@ and someone in the package's OWNERS file. If the license changed and requires a review, you should also ********************+emailremoved@.

  1. Add the new version (no one uses it)
  2. Switch everyone to use the new version
  3. Delete the old version

NOTE: Submitting CL#1 is committing to submitting CLs #2 and #3, or rolling back CL#1.

Why three CLs?

  • The separation of CLs #1 and #2 is to make the patch CL (#2) tiny. Patching a CL with large jars can be very slow at distributed offices with small pipes. If the first CL has already propagated via /path/to/.../build or *****, it is much less painful for them.
  • The separation of CLs #2 and #3 is to reduce the cost of an error by temporarily (1-2 days) leaving the old version. Teams can sometimes unbreak themselves with a temporary local BUILD file change if CL#2 is an issue (Explicitly list the old version as the first dep in the rule/binary that broke. Don't do this in common code!). This isn't a pretty state and should be rare. It is valuable when needed and it works.

The cost of creating the smaller CLs isn't much. This safety net matters more to teams outside of MTV since the odds go up that no one in MTV is around to roll back the CL for them.

CL #1: Add the new version

Follow the steps to import a new package, except you'll be importing only a new version instead.

You should already have OWNERS but you will still need emailremoved@ LGTM. If the license changed and you think it deserves a review, ********************+emailremoved@ to trigger it. Also always include someone in the package's OWNERS file for the approval.

CL #2: Updates BUILD files, sent out as a patch to be tested by all.

This CL will usually only touch BUILD files.

If code needs to be changed to use the new version, and the changes can be made while still on the old version (e.g. there were two ways to do X in v1, and only one in v2), they should be done in separate CLs prior to this CL being submitted. If the code changes need to be atomic with the version change, then they should be in this CL (rare).

  1. Change rule //third_party/java/<package>:<package> to point to the new version. The new rule should look like:

    java_library(
    name = "<packagename>",
    deps = [ "//third_party/java/<packagename>/<version>" ],
)

  2. Make any BUILD file anywhere in google3 that explicitly points at an old version point at the unversioned rule //third_party/java/<package>

    • No one should point at rule //third_party/java/<package>/<version> other than //third_party/java/<package>.
    • Once we have package visibility rules in all the version specific BUILD files, this will cease to be an issue.

  3. Try the CL against the global Presubmit presubmit queue

  4. Ask the package users to patch the CL and test it

    • If there are only a few users of this package, g4 mail the CL to those teams and ask them to test it on their project.
    • If there are many teams using the package, post to emailremoved@ asking teams to patch and test the CL.
    • Note that this is a NACK-based system. You should respond to objections, but silence is considered consent.

    • Sample note to emailremoved@

      If you don't use third_party/java/[PACKAGE], you can stop reading now.
What: Upgrading [PACKAGE] to [VERSION]
When: Submitting upgrade on [DATE 2 weeks from now] unless problems are reported.

ACTION REQUIRED
===============
If your project uses third_party/java/[PACKAGE], please patch and test CL [CL#2] this week.
If you have problems, please contact me immediately.

Why everyone needs to move
==========================
The transitive dependencies in the google3 build system requires that
everyone is on the same version of the libraries. If your transitive
build file dependencies lead to multiple versions, it is random which
one you, and any of your dependents, will get at runtime.

  5. Send the CL for review

    • If you had to change only a few BUILD files(3-4) outside of third-party java, get specific approvals from those teams.
    • If you had to change many, email emailremoved@ to ask for a review. Do not add emailremoved@ to the review list. Get a global approver assigned and then Cc emailremoved@ on the review.

  6. Create CL#3 to delete the old version so you don't forget.

  7. If all goes well, submit CL#2.

    • If you posted to emailremoved@, reply to the thread with the submitted CL#.

Make sure you and someone with approval powers to roll back the CL will be around at least three hours after submitting so it has time to go through a CB cycle on the larger projects.

If the new version does not work for any team, you and they need to resolve them. Depending on the changes, your need, and their availability, who does the work can vary. It is your job to ensure everything works with the new version prior to submitting this CL. It is not necessarily your job to do the work. Most teams are very cooperative and will make the changes if they understand what needs to be done. If the changes require expertise in the upgraded library, or you need them very quickly, it may be more efficient for you to do some or all of the work. Use common sense. After looking at what work is needed, you may decide that the upgrade to the new version isn't justified. In that case revert CL#2, and delete the new version you added in CL#1. You must not leave two versions in Piper.

If you have issues, email emailremoved@.

CL #3: Removes the old version

This is an easy CL. It should be submitted a day or two after CL #2. Just long enough to ensure CL #2 doesn't need to be rolled back.

cd third_party/java/<package>
g4 delete <old_version>/...
g4 mail

Submit this CL for approval to an owner of the specific third-party package.

Google modifications and additions to third_party Java code

If you're modifying a third party library, you'll want to use the preferred format of source based packages (under "Create the new package"), or the deprecated one of jar files plus source. But if you're enhancing it (i.e. adding new code, or writing tests), consider placing those enhancements in a com.google.thirdparty subpackage (in the normal //piper/.../google tree).

Jar files

Jar files for third-party code belong under //third_party/java or //googlemobile/third_party, and not under //java. Jars checked in under //java miss the appropriate license checks. This is not ok.

If you have a legitimate case to check in a jar file containing only Google owned code, add a JAR_REASON=reason you need a jar tag to your CL description. You may never use JAR_REASON for code not owned by Google.

Google3 source builds are fast enough, and the diamond dependency problem is ugly enough, that we want only source in google3 unless there is a really good reason.

If you have any questions about this policy, please email emailremoved@.

Other notes

Instructions for third_party/java reviewers.

What's up with java_src?

//piper/third_party/java_src exists because Blaze has particular rules for building targets with paths containing the name java, that can cause problems if you put source code in //piper/third_party/java. Each package in java_src is logically part of its related //piper/third_party/java package, and nothing except the associated //piper/third_party/java package should ever reference the java_src package.

What's up with fuzzers?

Fuzzing is a testing technique that feeds auto-generated inputs to a piece of target code in an attempt to crash the code. It's one of the most effective methods we have for finding bugs and vulnerabilities (see go/fuzzing-success). You can learn more about the benefits of fuzzing at go/why-fuzz, and specifically about java fuzzing at go/java-fuzzing.

  1. Fuzzing code should be put in fuzzer sub-directory of the package unless there are overriding concerns.
  2. Fuzzing code should be labeled as Google copyrighted (unencumbered) unless there are overriding concerns.
  3. This guidance is relatively new, so feel free to seek advice on emailremoved@.

It's the responsibility of the package OWNERS to upgrade them when the API of the fuzzed package changes, likely due to a version bump. In case of doubt/issues, ise-tps would be happy to give a hand.

Java is a registered trademark of Oracle and/or its affiliates.