From 2952c7bdb35ff20e201752c6270d4e53db47407d Mon Sep 17 00:00:00 2001 From: Rob Bavey Date: Fri, 11 Oct 2024 17:03:15 -0400 Subject: [PATCH 1/2] Add troubleshooting section for the date filter Relates: #16530 --- docs/static/troubleshoot/ts-date.asciidoc | 58 ++++++++++++++++++++ docs/static/troubleshoot/ts-plugins.asciidoc | 1 + 2 files changed, 59 insertions(+) create mode 100644 docs/static/troubleshoot/ts-date.asciidoc diff --git a/docs/static/troubleshoot/ts-date.asciidoc b/docs/static/troubleshoot/ts-date.asciidoc new file mode 100644 index 00000000000..539db473b4d --- /dev/null +++ b/docs/static/troubleshoot/ts-date.asciidoc @@ -0,0 +1,58 @@ +[[ts-date]] +==== Date Filter issues and solutions + +[discrete] +[[ts-date-locale]] +===== Date Parse Failures when using certain Locales + +*Symptoms* + +The {ls} Date Filter is unexpectedly unable to parse dates after upgrading to {ls} `8.15+`: + +A date filter configured to parse dates as follows: + +``` +date { locale => "en_gb" match => [ "[date]" , "yyyy MMM dd HH:mm:ss" ] target => "[parsed_date]" +``` + +when parsing the date `2024 Sep 30 11:11:11` would, prior to `8.15.0`, have created the following entry in the event: + +``` +"parsed_date" => 2024-09-30T11:11:11.000Z, +``` + +After `8.15.0`, this would have failed to parse with the following added instead: + +``` +"tags" => [ + [0] "_dateparsefailure" + ] +``` + +as it now expects the date to be `2024 Sept 30 11:11:11` for that specific locale. + + +*Background* + +Between JDK17 and JDK21, a change was made in the locale provider provided by the JVM internal. +JDK17 and JDK21 uses the https://cldr.unicode.org/[CLDR] locale provider by default, and between the release of JDK17 and JDK21, the version +of CLDR used by the default changed from version 37 to version 42. + +This change included an update that changed the abbreviated form of September from `Sep` to `Sept` in certain locales, including `en_GB` and `en_AU`. +This has caused some parse failures when there is a mismatch between the abbreviated form of date between Logstash and where the date was created. + +For example, if the systems running Logstash are running with either an `en_AU` or `en_GB locale`, they will + expect September dates to use the `Sept` abbreviated form when the format states `MMM`. +If the systems generating these dates are creating dates with a Sep abbreviation, parsing will fail. + +{ls} `8.15.0` upgraded the bundled JDK version to JDK21, which included the updated version of the `CLDR`, changing the +date parsing behavior for certain locales. + +** Mitigations** + +* Change the locale in the date filter to a locale that still uses `Sep` as the abbreviated form for September, such as `en_US`. +* Switch out the locale provider from CLDR to compat by setting `-Djava.locale.providers=COMPAT,SPI` in `jvm.options`. + Note that this will fail for anyone using JDK23 and above, which drops support for the `compat` locale provider +* Add filters to substitute `Sep` with `Sept` for all fields where the date is being parsed by the date filter. +Make sure these filters are *before* the date filter in the pipeline configuration. + diff --git a/docs/static/troubleshoot/ts-plugins.asciidoc b/docs/static/troubleshoot/ts-plugins.asciidoc index 839135c0511..ce30a60f0f3 100644 --- a/docs/static/troubleshoot/ts-plugins.asciidoc +++ b/docs/static/troubleshoot/ts-plugins.asciidoc @@ -3,3 +3,4 @@ include::ts-kafka.asciidoc[] include::ts-azure.asciidoc[] +include::ts-date.asciidoc[] From b104e48216ea3e3fb12a8e3f1a06edab6be76c5a Mon Sep 17 00:00:00 2001 From: Rob Bavey Date: Mon, 4 Nov 2024 09:12:47 -0500 Subject: [PATCH 2/2] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: João Duarte --- docs/static/troubleshoot/ts-date.asciidoc | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/static/troubleshoot/ts-date.asciidoc b/docs/static/troubleshoot/ts-date.asciidoc index 539db473b4d..71cc732df45 100644 --- a/docs/static/troubleshoot/ts-date.asciidoc +++ b/docs/static/troubleshoot/ts-date.asciidoc @@ -7,7 +7,7 @@ *Symptoms* -The {ls} Date Filter is unexpectedly unable to parse dates after upgrading to {ls} `8.15+`: +The {ls} Date Filter is unable to parse certain dates after upgrading to {ls} `8.15+`: A date filter configured to parse dates as follows: @@ -38,8 +38,8 @@ Between JDK17 and JDK21, a change was made in the locale provider provided by th JDK17 and JDK21 uses the https://cldr.unicode.org/[CLDR] locale provider by default, and between the release of JDK17 and JDK21, the version of CLDR used by the default changed from version 37 to version 42. -This change included an update that changed the abbreviated form of September from `Sep` to `Sept` in certain locales, including `en_GB` and `en_AU`. -This has caused some parse failures when there is a mismatch between the abbreviated form of date between Logstash and where the date was created. +To accommodate CLDR 38 in the JDK, https://github.com/openjdk/jdk/pull/1279/files#diff-97210acd6f77c4f4979c43445d60ba1c369f058230e41177dceca697800b1fa2R116[openjdk/jdk#1279] changed the abbreviated form of September from `Sep` to `Sept` in certain locales, including `en_GB` and `en_AU`. +This caused parse failures when there is a mismatch in the abbreviated form of date between Logstash and where the date was created. For example, if the systems running Logstash are running with either an `en_AU` or `en_GB locale`, they will expect September dates to use the `Sept` abbreviated form when the format states `MMM`. @@ -52,7 +52,6 @@ date parsing behavior for certain locales. * Change the locale in the date filter to a locale that still uses `Sep` as the abbreviated form for September, such as `en_US`. * Switch out the locale provider from CLDR to compat by setting `-Djava.locale.providers=COMPAT,SPI` in `jvm.options`. - Note that this will fail for anyone using JDK23 and above, which drops support for the `compat` locale provider -* Add filters to substitute `Sep` with `Sept` for all fields where the date is being parsed by the date filter. -Make sure these filters are *before* the date filter in the pipeline configuration. +Note that JDK23 drops support for the `compat` locale provider. A solutions is to add mutate filters to substitute `Sep` with `Sept` for all fields where the date is being parsed by the date filter. +Make sure these filters are placed *before* the date filter in the pipeline configuration.