Turns out you can time travel using Java.

Background

I work in a Security Operation Centre (SOC) as a System Security Engineer. Over the years, we've helped built bespoke in-house software to take in security alerts from our client's environments for triage, and starting on Boxing Day 2021, our team noticed something bizarre. All logs stopped being received by our Elasticsearch instance.

The team responded within the first 24 hours, with the initial belief that this was an issue with security alerts not being able to be created within Elasticsearch, as the number of events being seen started declining rapidly to 0. With no logs that would indicate any failure in any of our software querying or pushing data to Elastic, and Elastic itself not showing any problems of indexing and ingesting new data, we were left scratching heads over what might have gone awry.

What went wrong?

Our in-house systems – written entirely using Scala by an incredibly talented Software Engineering team – parse and present Security events coming from a variety of different SIEMs, security applications and data sources, in order to be triaged by our Analyst team.

We rely on these services to enhance, correlate and relate different security alerts/incidents together. This process requires a database of accurate temporal documents in order to find any similarities between any two given events.

One of our members from the Security Analyst team also raised to us that we had not seen any alerts for a lot of our clients in an unexpectedly long timeframe, and after checking any Managed SIEM services we provide to our clients, and finding plenty of alerts being triggered and making their way to our security case management system, I knew this would be triggered by the Elastic event dropoff, as we rely heavily on Elastic to provide this sanity check.

One of the first indicators was that this incident happened directly on midnight UTC (11am on +11 UTC) between Christmas Day and Boxing day. We initially did a lot of inspection to see if we had any certificates within our alert flow that we had not come across before. This didn't turn out to be the case.

Inspecting some of the alerts that had been raised successfully for triage, I noticed something odd, all security alerts we're being raised in 2022, Noting that I had not yet rang in the new year (or at least in a post-Christmas Coma, I didn't think I had yet) and having to do a double take when observing "Dec 2022", it became obvious that this was no longer an Elastic problem, but instead a time-based problem located somewhere in the alerting pipeline.

Looking at Elastic events in 2022, lo and behold, we find our missing security events & alerts.

This actually took a non-trivial time to find as Elastic's default time picker doesn't provide many any time-frames to search the future.

I ended up hunting down whether a rouge NTP bug or upstream peer had somehow caused the system time to diverge:

$ date
Sun Dec 26 10:19:25 AEDT 2021

Huh, noting nothing on the NTP / System Time side of things (in hindsight, all TLS certificates would've been invalid, among other errors if system time jumped ahead by a full year), maybe the Java DateTime takes off the hardware clock instead?

$ sudo hwclock --show
Sun Dec 26 2021 10:21:10 AEDT 2021 -0.054098 seconds

No signs of problems here either. I can see that the year bug is affecting all clients using a variety of different SIEM technologies. What was interesting is that because of this, the problem was not located upstream and was squarely located within our in-house software.

Building the Proof of Concept

From here it seemed fairly direct where to find all locations where we parse / use any date time libraries within our code, which allowed me to build the following Proof of Concept to submit the bug report to our Software Engineering team.

I found the spot of concern, using Java's DateTimeFormatter to parse and present timestamps from SIEMs, and I wanted to trial out using two different forms of year presentation.

I created two formatters, one using YYYY (as we had been in our parsers), and one using yyyy.

import java.time.format.DateTimeFormatter;
import java.time.LocalDateTime;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.time.ZonedDateTime;

public class ProofOfConcept {
	public static void main(String[] args) {
		DateTimeFormatter formatCapitalY = DateTimeFormatter.ofPattern("YYYY-MM-dd'T'HH:mm:ss.SSSXXX'['VV']'");
		DateTimeFormatter formatLowercaseY = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSXXX'['VV']'");

		LocalDateTime now = LocalDateTime.now();

		System.out.println(formatCapitalY.format(now));
		System.out.println(formatLowercaseY.format(now));
	}
}

Running the Proof of Concept yields the following:

$ javac ProofOfConcept.java

$ java ProofOfConcept
2022-12-26T11:30:01.469+11:00[Australia/Sydney]
2021-12-26T11:30:01.469+11:00[Australia/Sydney]

Ah hah! 🎉 We found the bug, but why would this be happening? It seems to be fine as of time of writing the article...

$ javac ProofOfConcept.java

$ java ProofOfConcept
2022-01-25T17:47:01.469+11:00[Australia/Sydney]
2022-01-25T17:47:01.469+11:00[Australia/Sydney]

Introducing ISO 8601

The Java documentation on DateTimeFormatter indicates that YYYY is the format of a "week-based-year"... Ok, so we should be using the Gregorian calendar year version which is denoted by yyyy. A simple enough fix. But what is a week-based-year?

According to https://help.tableau.com/current/pro/desktop/en-us/dates_calendar.htm, a week based year differs from a Gregorian calendar year as:

ISO-8601 years always start on the first Monday closest to January 1st. This may mean that the year does not start until January 4th in some cases, or may start in late December in others. Gregorian calendars always start the year on the 1st of January. This can cause some discrepancy in years between the two calendar systems when very close to the beginning of January.

Well hang on a sec, that doesn't make sense. The 3^rd of Jan 2022 would be the closest date to the 1^st. Why is Java reporting the jump to the 26th of December? What does Wikipedia say?

... [YYYY] indicates the ISO week-numbering year which is slightly different from the traditional Gregorian calendar year. ...

Ok, it goes on to mention "several mutually equivalent and compatible descriptions" of week 01

1. the week with the first business day in the starting year (considering that Saturdays, Sundays and 1st January are non-working days) ❌
2. the week with the starting year's first Thursday in it (the formal ISO definition) ❌
3. the week with 4 January in it ❌
4. the first week with the majority (four or more) of its days in the starting year, and ❌
5. the week starting with the Monday in the period 29 December - 4 January. ❌

None of these explanations fit our description...

Java's Implementation of "Week Of Year"

I found the answer on a StackOverflow question filed in 2011.

Y returns 2012 while y returns 2011 in SimpleDateFormat

I wonder why ‘Y’ returns 2012 while ‘y’ returns 2011 in SimpleDateFormat: System.out.println(new SimpleDateFormat(“Y”).format(new Date())); // prints 2012 System.out.println(new SimpleDateFormat(”...

Stack OverflowEng.Fouad

Quoting the original source, it first goes on to explain.

Values calculated for the WEEK_OF_YEAR field range from 1 to 53. The first week of a calendar year is the earliest seven day period starting on getFirstDayOfWeek() that contains at least getMinimalDaysInFirstWeek() days from that year.

Well, if we have getMinimalDaysInFirstWeek() being 1, we should have a winner.

// create a calendar
Calendar cal = Calendar.getInstance();

// get what the minimal days required in the first week of the year 
int i = cal.getMinimalDaysInFirstWeek();

// print the result
System.out.println("Minimal days required: " + i);

> Minimal days required: 1

It thus depends on the values of getMinimalDaysInFirstWeek(), getFirstDayOfWeek(), and the day of the week of January 1.

The week determination is compatible with the ISO 8601 standard when getFirstDayOfWeek() is MONDAY and getMinimalDaysInFirstWeek() is 4, which values are used in locales where the standard is preferred.

These values can explicitly be set by calling setFirstDayOfWeek() and setMinimalDaysInFirstWeek().

So we reach the end of the journey. Software Engineering received the ticket, and set it to P5 and Due Date: Dec 2022.

See you all later this year!