This describes Java-specific guidance for checking code into //piper/third_party/java.
IMPORTANT: Read go/thirdparty first. For //piper/third_party/java specific questions, email emailremoved@.
NOTE: The //piper/third_party/java rules are only required for google3 code. If you are submitting code into a different depot, you should consider having //piper/third_party/java mirrored into your depot (see go/piper-task-bug), which gives you the benefit of all the maintenance work that goes into the primary depot, but this is not required.
Comments from experience
Most open source code is very good about being backwards compatible. Upgrading to new versions is often no more than giving teams a chance to verify that running against the new version still works.
IMPORTANT: If something doesn't work after an upgrade, rollback and then figure it out. The cost of broken builds across the company is extremely high.
Individual teams may say, "We don't want to change".
- If a team can justify why the upgrade can't be done or needs to be delayed, listen.
- If they can't justify it, tough. Going through these upgrades is part of the price of using the google3 build system. For most upgrades, two weeks should be plenty of time for a team to find time to test the change and possibly make minor changes to their code.
If you are not sure what to do at any point, email emailremoved@
Adding a new library
Check your preconditions
- Verify the library is not already there under a slightly different name. The current naming is unfortunately often not consistent.
- Verify there isn't already another similar library present. For example, there are many json parsers or logging frameworks. Please use one of the ones already present.
- Ensure all libraries your new package depends on are present at an
acceptable version. If your new package depends on other third party
packages, do not submit those additional jars within your new package.
This will lead to broken builds (see go/oneversion). Instead:
- Depend on existing rules //piper/third_party/java for that supporting package.
- If the exact version needed is not present, you must either use the existing version, or upgrade all of google to use the new version you require.
- If the supporting package does not exist in //piper/third_party/java, add it in a separate changelist as a precondition of your current changelist.
Take the third_party/java owners oath.
# TODO: Hold up your right hand and repeat this. # # WE THE OWNERS HEREBY PROMISE to ensure that there will only be one version # of this project in google3 at a time. # # If at any time we g4 approve a new version, we promise to work with the # submitter to ensure that: # # 1) All of Google can move to the new version in one CL. # # 2) If a new version will break other teams ... # a) We will not submit until we find a way to do so without breaking them. # b) If we can update all but 1-2 teams without breaking those other teams, # and we really need this version now, we can submit providing we commit # to being back on one version Google wide within TWO WEEKS. # # 3) The old version will be deleted within two weeks of submitting # the new version. # # 4) Any dependencies of this package will be separate third_party/java # entries. (No /lib directories or equivalent). # # 5) We will always depend on the unversioned rule for dependencies. # (e.g. //third_party/java/foo, not //third_party/java/foo/v1_3) # # IF THE UPGRADE SUBMITTER DOES NOT FOLLOW THROUGH, WE PROMISE THAT WE # WILL EITHER FINISH FOR THEM OR IMMEDIATELY DELETE THE NEW VERSION.
Create the new package
There are three common formats that you will find for Java projects in //third_party.
NOTE: Starting in late 2019, third party Java usage at Google has standardized on moving to source based builds by default, previously known as "format 3." This is now described as the "preferred format" below.
In each case:
- Copy the appropriate templates, and fix the TODO's in each copied file. Please remember to remove the TODO lines after you address them.
- In some cases there is a
README.mdfile to provide additional requirements and guidance.
- The initial changelist should contain a single new third-party package and nothing else (treating any //piper/third_party/java package and the corresponding //piper/third_party/java_src package as a "single package" in such cases). This submits the pristine sources and the go/thirdparty "metadata" together in a single, atomic changelist.
- The go/thirdparty
METADATAfile belongs in the
java_srctree. (Legacy libraries may have it in the
(All of the commands below assume you start in the
Preferred format: Source based packages
MY_NEW_PACKAGE=my_new_package mkdir -p third_party/java_src/$MY_NEW_PACKAGE third_party/java/$MY_NEW_PACKAGE cp -R /path/to/.../* third_party/java_src/$MY_NEW_PACKAGE/ cp -R /path/to/.../* third_party/java/$MY_NEW_PACKAGE/ chmod 640 $(find third_party/java/$MY_NEW_PACKAGE third_party/java_src/$MY_NEW_PACKAGE -type f) # ensure files found by g4 nothave chmod +w third_party/java/$MY_NEW_PACKAGE third_party/java_src/$MY_NEW_PACKAGE
Although the source code is checked in to the
java_src package and users
compile it at build time, a //piper/third_party/java package is still required,
as users only ever depend on packages in //piper/third_party/java.
To help facilitate regular upgrades as part of the
maintenance requirements for packages in //third_party,
you may find it useful to automate imports from upstream sources of truth where
possible using copybara. The sample package directories
include an empty
copy.bara.sky that you can modify, but copybara can also
synthesize one based on the contents of your
copybara info third_party/java_src/$MY_NEW_PACKAGE/METADATA --metadata-generate-common-libraries --metadata-disable-url-translation --verbose
You can apply local changes by defining Copybara transformations. and can run the import with the following command:
g4 change # to create a change copybara --init-history --destination-cl=$(/path/to/.../vcstool pending-change-number) third_party/java_src/$MY_NEW_PACKAGE/copy.bara.sky
To make regular upgrades even easier, if you register your configuration with Copybara-as-a-Service, upgrade changelists will be automatically generated for review when the conditions you specify are met.
NOTE: Allowing local modifications for packages primarily developed elsewhere may cause pain long-term when upgrading to a new version. It might be worth insisting that issues are fixed upstream and the fixes imported using copybara. See bug ...nt2 for context.
Reference documentation for the other, deprecated formats remains below for
convenience. Use of these formats requires an exception, which is verified by
BannedExtension check of go/compliancelint. A list of active exceptions
can be found at go/empty-jars-exceptions.
By Exception Only / Deprecated Format 1: Jar files
IMPORTANT: This format is considered deprecated and should be used only when source code is not available. To submit a new package, or upgrade an existing package where only a .jar is available, fill out the exception template at go/third-party-removed. The component is managed by blunderbuss and will be reviewed by ISE-TPS with a target SLO of under 4 business days for response.
MY_NEW_PACKAGE=my_new_package cp -R /path/to/.../* third_party/java/$MY_NEW_PACKAGE chmod 640 $(find third_party/java/$MY_NEW_PACKAGE -type f) # ensure files found by g4 nothave chmod +w third_party/java/$MY_NEW_PACKAGE
In this format, you never submit the expanded source. You build locally and only
submit the updated jars. Be sure to include a
source is available, and read the
New .jar-only packages should have restricted visibility and an appropriate Category field set in the METADATA file, determined as part of the exception consultation with ISE-TPS.
By Exception Only / Deprecated Format 2: Jar files plus source
This format requires extra work over Deprecated Format #1 with little to no benefit, so most users that want to commit the source should use the preferred source-only format. Rarely, the overhead of compiling the package may be large or the tooling cumbersome, in which case this format could be the best fit. To submit a new package in this format, fill out the exception template at go/third-party-removed. The component is managed by blunderbuss and will be reviewed by ISE-TPS with a target SLO of under 4 business days for response.
MY_NEW_PACKAGE=my_new_package cp -R /path/to/.../SAMPLE_JAVA_SRC_PACKAGE third_party/java_src/$MY_NEW_PACKAGE cp -R /path/to/.../SAMPLE_NEW_PACKAGE third_party/java/$MY_NEW_PACKAGE chmod 640 $(find third_party/java/$MY_NEW_PACKAGE third_party/java_src/$MY_NEW_PACKAGE -type f) # ensure files found by g4 nothave chmod +w third_party/java/$MY_NEW_PACKAGE third_party/java_src/$MY_NEW_PACKAGE
In this format, you submit both the source code (in the
java_src package) and
compiled jars (in the java package). Be sure to include a
-sources.jar in the java package as well.
Warnings and errors in //third_party code
Google's Java compiler adds additional bug checkers using go/errorprone, so third-party code may not compile with Blaze. error-prone team tries to ensure that our additional checkers detect high-value bugs, so we strongly encourage you to fix any bugs it points out and either maintain a local patch or push the changes upstream. However, we understand that in some cases this may not be possible, and you will need to disable the checks. There are two ways to do this:
You can disable a specific check by passing an additional java compiler option (go/be#java_library.javacopts),
"-Xep:checkname:OFF", to the
checknameis given in brackets in the error message. For example, given the error message:
[LongLiteralLowerCaseSuffix] Prefer '
L' to '
l' for the suffix to long literals
you would pass the compiler option
"-Xep:LongLiteralLowerCaseSuffix:OFF"to disable it. You can disable multiple checks by passing a separate flag for each one. Note that only certain checks are allowed to be disabled; you will get an error message if you try to disable a non-disableable check.
You can disable all suppressible error-prone checks by passing the java compiler option (go/be#java_library.javacopts)
The initial changelist should not include any local modifications.
When making local changes to the third-party source code, a "pristine" copy of
the code, together with the necessary Google-specific files, must be submitted
in the initial changelist . Local modifications must then be made in a follow-up
changelist. This subsequent changelist can be reviewed by the custodial owners
of the new package (those in the newly created
OWNERS file), unless there are
license or security considerations associated with the modifications that
require review by others.
If you are using jar-only format, where we never check in the expanded source,
please do submit an unmodified
submitting a modified
x-sources.jar and patch file.
x-sources.jar must always have the same directory
x.jar; this is not considered a "local modification".)
When you are ready to submit
blaze build third_party/java/$MY_NEW_PACKAGE:all to look for typos.
Send the CL for review:
- add to
- another new owner
emailremoved@(for legal/license issues),
emailremoved@(for everything else).
Both groups must LGTM the CL. The second to do so will approve.
Resolving conflicts between multiple versions
Cleaning up conflicts, including existing multiple versions, is very similar to doing an upgrade.
If the conflict is caused by two versions of the same package in third party java:
- Please upgrade everyone to the more recent version and remove the older version. Instructions—start at upgrade CL#2 below.
If the conflict is caused by classes embedded in another jar:
- Remove all embedded dependencies from the jar and depend on the official
versions checked into third_party/java instead. For example, if the selenium
jar also contained a version of junit, you would remove the junit classes
from the selenium jar and add //piper/third_party/java/junit to the
//piper/third_party/java/selenium:selenium rules dependencies. Script
/path/to/.../remove_files_from_jardoes the heavy lifting for this.
- If the version of the conflicting classes you need is not in
third_party/java, and the version you need is older than the version submitted in
- Try using the newer version. Often it just works
- If the code requiring the older third party library is Google code, please upgrade it to work with the newer version already in //piper/third_party/java.
- If the dependent code is another third party library, please find a newer version that will work and follow the upgrade procedures below to move everyone to the newer version.
- If the version of the conflicting classes you need is not in
//piper/third_party/java, and the version you need is newer than the version
already in //piper/third_party/java:
- Try using the older version. Sometimes you get lucky and it just works.
- If that doesn't work, please follow the upgrade procedures at to add the newer version and move everyone to it.
Upgrading an existing library (Note: There is no "add new version.")
The ground rules
When you add a new version, you are committing to upgrading all of Google to use the new version in one CL. You may not use the new version in your project beyond compatibility testing before this. The old version should be deleted as soon as it looks like the conversion CL will not need to be rolled back. The multiple version layout is only to ease the testing of the new version prior to converting all of Google. You may not just introduce a new version. See go/oneversion for an explanation why.
Man, that sounds harsh, doesn't it? Actually, it is a whole lot easier than you think. In practice, we find that if you follow the procedure outlined below, most upgrades take 4-8 hours of your time, spread across several days or weeks; the (rare) really, really ugly ones requiring chains of upgrades can take 2 weeks of time over 4-6 weeks. It turns out that having one person pay this cost and coordinate for everyone is a bargain to the company. Do you know of any other company where you could do this in less than a day of your time? How much time would each project need to spend to do this on their own? How much would be lost in version conflicts in the meantime?
The sole exceptions to this rule is JDBC drivers under //piper/third_party/java/jdbc. For JDBC, moving databases is such a hard task that the decision was made several years ago that all database dependencies can only be specified on the binary build target. You may not specify them as a dependency for a library build target. This avoids almost all of the version skew issues. (If your binary has a data dependency on another binary, it can still bite you.)
The three CLs
If the third_party library you are upgrading has very few users, you may do all of these in a single CL. Submit this CL for review to emailremoved@ and someone in the package's OWNERS file. If the license changed and requires a review, you should also ********************+emailremoved@.
- Add the new version (no one uses it)
- Switch everyone to use the new version
- Delete the old version
NOTE: Submitting CL#1 is committing to submitting CLs #2 and #3, or rolling back CL#1.
Why three CLs?
- The separation of CLs #1 and #2 is to make the patch CL (#2) tiny. Patching a CL with large jars can be very slow at distributed offices with small pipes. If the first CL has already propagated via /path/to/.../build or *****, it is much less painful for them.
- The separation of CLs #2 and #3 is to reduce the cost of an error by
temporarily (1-2 days) leaving the old version. Teams can sometimes unbreak
themselves with a temporary local
BUILDfile change if CL#2 is an issue (Explicitly list the old version as the first dep in the rule/binary that broke. Don't do this in common code!). This isn't a pretty state and should be rare. It is valuable when needed and it works.
The cost of creating the smaller CLs isn't much. This safety net matters more to teams outside of MTV since the odds go up that no one in MTV is around to roll back the CL for them.
CL #1: Add the new version
Follow the steps to import a new package, except you'll be importing only a new version instead.
You should already have OWNERS but you will still need emailremoved@ LGTM. If the license changed and you think it deserves a review, ********************+emailremoved@ to trigger it. Also always include someone in the package's OWNERS file for the approval.
CL #2: Updates BUILD files, sent out as a patch to be tested by all.
This CL will usually only touch
If code needs to be changed to use the new version, and the changes can be made
while still on the old version (e.g. there were two ways to do X in
only one in
v2), they should be done in separate CLs prior to this CL being
submitted. If the code changes need to be atomic with the version change, then
they should be in this CL (rare).
//third_party/java/<package>:<package>to point to the new version. The new rule should look like:
java_library( name = "<packagename>", deps = [ "//third_party/java/<packagename>/<version>" ], )
BUILDfile anywhere in google3 that explicitly points at an old version point at the unversioned rule
- No one should point at rule
- Once we have package visibility rules in all the version specific BUILD files, this will cease to be an issue.
- No one should point at rule
Try the CL against the global Presubmit presubmit queue
Ask the package users to patch the CL and test it
- If there are only a few users of this package, g4 mail the CL to those teams and ask them to test it on their project.
- If there are many teams using the package, post to emailremoved@ asking teams to patch and test the CL.
- Note that this is a NACK-based system. You should respond to objections, but silence is considered consent.
Sample note to emailremoved@
If you don't use third_party/java/[PACKAGE], you can stop reading now. What: Upgrading [PACKAGE] to [VERSION] When: Submitting upgrade on [DATE 2 weeks from now] unless problems are reported. ACTION REQUIRED =============== If your project uses third_party/java/[PACKAGE], please patch and test CL [CL#2] this week. If you have problems, please contact me immediately. Why everyone needs to move ========================== The transitive dependencies in the google3 build system requires that everyone is on the same version of the libraries. If your transitive build file dependencies lead to multiple versions, it is random which one you, and any of your dependents, will get at runtime.
Send the CL for review
- If you had to change only a few
BUILDfiles(3-4) outside of third-party java, get specific approvals from those teams.
- If you had to change many, email emailremoved@ to ask for a review. Do not add emailremoved@ to the review list. Get a global approver assigned and then Cc emailremoved@ on the review.
- If you had to change only a few
Create CL#3 to delete the old version so you don't forget.
If all goes well, submit CL#2.
- If you posted to emailremoved@, reply to the thread with the submitted CL#.
Make sure you and someone with approval powers to roll back the CL will be around at least three hours after submitting so it has time to go through a CB cycle on the larger projects.
If the new version does not work for any team, you and they need to resolve them. Depending on the changes, your need, and their availability, who does the work can vary. It is your job to ensure everything works with the new version prior to submitting this CL. It is not necessarily your job to do the work. Most teams are very cooperative and will make the changes if they understand what needs to be done. If the changes require expertise in the upgraded library, or you need them very quickly, it may be more efficient for you to do some or all of the work. Use common sense. After looking at what work is needed, you may decide that the upgrade to the new version isn't justified. In that case revert CL#2, and delete the new version you added in CL#1. You must not leave two versions in Piper.
If you have issues, email emailremoved@.
CL #3: Removes the old version
This is an easy CL. It should be submitted a day or two after CL #2. Just long enough to ensure CL #2 doesn't need to be rolled back.
cd third_party/java/<package> g4 delete <old_version>/... g4 mail
Submit this CL for approval to an owner of the specific third-party package.
Google modifications and additions to third_party Java code
If you're modifying a third party library, you'll want to use the preferred
format of source based packages (under "Create the new package"), or the
deprecated one of jar files plus source. But if you're enhancing it (i.e. adding
new code, or writing tests), consider placing those enhancements in a
com.google.thirdparty subpackage (in the normal //piper/.../google tree).
Jar files for third-party code belong under
//googlemobile/third_party, and not under
//java. Jars checked in under
//java miss the appropriate license checks. This is not ok.
If you have a legitimate case to check in a jar file
containing only Google owned code, add a
JAR_REASON=reason you need a jar tag
to your CL description. You may never use JAR_REASON for code not owned by
Google3 source builds are fast enough, and the diamond dependency problem is ugly enough, that we want only source in google3 unless there is a really good reason.
If you have any questions about this policy, please email emailremoved@.
What's up with java_src?
//piper/third_party/java_src exists because Blaze has particular rules for
building targets with paths containing the name
java, that can cause problems
if you put source code in //piper/third_party/java. Each package in
is logically part of its related //piper/third_party/java package, and nothing
except the associated //piper/third_party/java package should ever reference the
What's up with fuzzers?
Fuzzing is a testing technique that feeds auto-generated inputs to a piece of target code in an attempt to crash the code. It's one of the most effective methods we have for finding bugs and vulnerabilities (see go/fuzzing-success). You can learn more about the benefits of fuzzing at go/why-fuzz, and specifically about java fuzzing at go/java-fuzzing.
- Fuzzing code should be put in fuzzer sub-directory of the package unless there are overriding concerns.
- Fuzzing code should be labeled as Google copyrighted (unencumbered) unless there are overriding concerns.
- This guidance is relatively new, so feel free to seek advice on emailremoved@.
It's the responsibility of the package OWNERS to upgrade them when the API of the fuzzed package changes, likely due to a version bump. In case of doubt/issues, ise-tps would be happy to give a hand.
Java is a registered trademark of Oracle and/or its affiliates.