Corrupted Database

[gustavohellwig]

Hi,
I'm trying to update the mmdb database, as there is missing data in some ips, as Graylog needs those information to treat the logs.
However, after I update the mmdb, the graylog lookup table doesn't work.
The result from the test is this:

{
  "single_value": null,
  "multi_value": null,
  "string_list_value": null,
  "has_error": false,
  "ttl": 9223372036854776000
}

normally, it would be this:

{
  "single_value": "TZ",
  "multi_value": {
    "continent": {
      "code": "AF",
      "geoname_id": 6255146,
      "names": {
        "de": "Afrika",
        "ru": "Африка",
        "pt-BR": "África",
        "ja": "アフリカ",
        "en": "Africa",
        "fr": "Afrique",
        "zh-CN": "非洲",
        "es": "África"
      }
    },
    "country": {
      "confidence": null,
      "geoname_id": 149590,
      "is_in_european_union": false,
      "iso_code": "TZ",
      "names": {
        "de": "Tansania",
        "ru": "Танзания",
        "pt-BR": "Tanzânia",
        "ja": "タンザニア連合共和国",
        "en": "Tanzania",
        "fr": "Tanzanie",
        "zh-CN": "坦桑尼亚",
        "es": "Tanzania"
      }
    },
    "traits": {
      "autonomous_system_number": null,
      "autonomous_system_organization": null,
      "connection_type": null,
      "domain": null,
      "ip_address": "41.59.208.37",
      "is_anonymous": false,
      "is_anonymous_proxy": false,
      "is_anonymous_vpn": false,
      "is_anycast": false,
      "is_hosting_provider": false,
      "is_legitimate_proxy": false,
      "is_public_proxy": false,
      "is_residential_proxy": false,
      "is_satellite_provider": false,
      "is_tor_exit_node": false,
      "isp": null,
      "mobile_country_code": null,
      "mobile_network_code": null,
      "network": "41.59.0.0/16",
      "organization": null,
      "user_type": null,
      "user_count": null,
      "static_ip_score": null
    },
    "represented_country": {
      "confidence": null,
      "geoname_id": null,
      "is_in_european_union": false,
      "iso_code": null,
      "names": {},
      "type": null
    },
    "registered_country": {
      "confidence": null,
      "geoname_id": 149590,
      "is_in_european_union": false,
      "iso_code": "TZ",
      "names": {
        "de": "Tansania",
        "ru": "Танзания",
        "pt-BR": "Tanzânia",
        "ja": "タンザニア連合共和国",
        "en": "Tanzania",
        "fr": "Tanzanie",
        "zh-CN": "坦桑尼亚",
        "es": "Tanzania"
      }
    }
  },
  "string_list_value": null,
  "has_error": false,
  "ttl": 9223372036854776000
}

what I have done:

1. installed the app:

wget "https://github.com/ipinfo/mmdbctl/releases/download/mmdbctl-1.4.4/mmdbctl_1.4.4_linux_amd64.tar.gz" && tar xvf mmdbctl_1.4.4_linux_amd64.tar.gz && rm mmdbctl_1.4.4_linux_amd64.tar.gz && mv mmdbctl_1.4.4_linux_amd64 /usr/local/bin/mmdbctl
sudo apt-get update && sudo apt-get install jq

2. Exported:
   mmdbctl export --in ${importfrom} --out ${exportto}

3. Changed the content using python:

import json

input_file = '/opt/scripts/tmp/GeoLite2-City.json'
output_file = '/opt/scripts/tmp/GeoLite2-City-Changes.json'

def add_subdivisions_to_json_line(json_line):
    data = json.loads(json_line)
    original_data = data.copy()  # Make a copy of the original data for comparison

    if 'subdivisions' not in data:
        country_info = data.get('country', {})
        if country_info:
            subdivisions = [{
                "geoname_id": country_info.get("geoname_id"),
                "iso_code": country_info.get("iso_code"),
                "names": {"en": country_info["names"]["en"]}
            }]
            data['subdivisions'] = subdivisions

    return data, data != original_data  # Return both the data and a flag indicating if a change was made

with open(input_file, 'r') as infile, open(output_file, 'w') as outfile:
    for line in infile:
        updated_data, is_changed = add_subdivisions_to_json_line(line)
        if is_changed:
            updated_line = json.dumps(updated_data)
            outfile.write(updated_line + '\n')

print(f"Processed file saved as {output_file}")

4. Imported the changes to the mmdb:
   mmdbctl import --disable-metadata-pointers false --in ${updatedjson} --out ${mmdbbkp}

5. Tests
   mmdbctl metadata GeoLite2-City-updated.mmdb

• Binary Format 2.0
• Database Type ipinfo GeoLite2-City-updated.mmdb
• IP Version 6
• Record Size 32
• Node Count 3971781
• Description
  en ipinfo GeoLite2-City-updated.mmdb
• Languages en
• Build Epoch 1721934123

mmdbctl metadata GeoLite2-City.mmdb

• Binary Format 2.0
• Database Type GeoLite2-City
• IP Version 6
• Record Size 28
• Node Count 3971880
• Description
  en GeoLite2City database
• Languages de, en, es, fr, ja, pt-BR, ru, zh-CN
• Build Epoch 1721747951

mmdbctl read -f json-pretty 41.59.208.37 GeoLite2-City.mmdb # Original File
{
"continent": {
"code": "AF",
"geoname_id": 6255146,
"names": {
"de": "Afrika",
"en": "Africa",
"es": "África",
"fr": "Afrique",
"ja": "アフリカ",
"pt-BR": "África",
"ru": "Африка",
"zh-CN": "非洲"
}
},
"country": {
"geoname_id": 149590,
"iso_code": "TZ",
"names": {
"de": "Tansania",
"en": "Tanzania",
"es": "Tanzania",
"fr": "Tanzanie",
"ja": "タンザニア連合共和国",
"pt-BR": "Tanzânia",
"ru": "Танзания",
"zh-CN": "坦桑尼亚"
}
},
"location": {
"accuracy_radius": 1000,
"latitude": -6.8227,
"longitude": 39.2936,
"time_zone": "Africa/Dar_es_Salaam"
},
"registered_country": {
"geoname_id": 149590,
"iso_code": "TZ",
"names": {
"de": "Tansania",
"en": "Tanzania",
"es": "Tanzania",
"fr": "Tanzanie",
"ja": "タンザニア連合共和国",
"pt-BR": "Tanzânia",
"ru": "Танзания",
"zh-CN": "坦桑尼亚"
}
}
}

mmdbctl read -f json-pretty 41.59.208.37 GeoLite2-City-updated.mmdb # New File
{
"continent": {
"code": "AF",
"geoname_id": 6255146,
"names": {
"de": "Afrika",
"en": "Africa",
"es": "África",
"fr": "Afrique",
"ja": "アフリカ",
"pt-BR": "África",
"ru": "Африка",
"zh-CN": "非洲"
}
},
"country": {
"geoname_id": 149590,
"iso_code": "TZ",
"names": {
"de": "Tansania",
"en": "Tanzania",
"es": "Tanzania",
"fr": "Tanzanie",
"ja": "タンザニア連合共和国",
"pt-BR": "Tanzânia",
"ru": "Танзания",
"zh-CN": "坦桑尼亚"
}
},
"location": {
"accuracy_radius": 1000,
"latitude": -6.8227,
"longitude": 39.2936,
"time_zone": "Africa/Dar_es_Salaam"
},
"network": "41.59.208.0/20",
"registered_country": {
"geoname_id": 149590,
"iso_code": "TZ",
"names": {
"de": "Tansania",
"en": "Tanzania",
"es": "Tanzania",
"fr": "Tanzanie",
"ja": "タンザニア連合共和国",
"pt-BR": "Tanzânia",
"ru": "Танзания",
"zh-CN": "坦桑尼亚"
}
},
"subdivisions": [
{
"geoname_id": 149590,
"iso_code": "TZ",
"names": {
"en": "Tanzania"
}
}
]
}

Graylog log:

WARN [MaxmindDataAdapter] Unable to look up city data for IP address /123.185.49.44, returning empty result.
java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "parameters[index]" is null
at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:450) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decodeMap(Decoder.java:341) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decode(Decoder.java:89) ~[graylog.jar:?]
at com.maxmind.db.NoCache.get(NoCache.java:17) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decode(Decoder.java:116) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decodeMapIntoObject(Decoder.java:434) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decodeMap(Decoder.java:341) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decodeByType(Decoder.java:162) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[graylog.jar:?]
at com.maxmind.db.Decoder.decode(Decoder.java:76) ~[graylog.jar:?]
at com.maxmind.db.Reader.resolveDataPointer(Reader.java:411) ~[graylog.jar:?]
at com.maxmind.db.Reader.getRecord(Reader.java:185) ~[graylog.jar:?]
at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:280) ~[graylog.jar:?]
at com.maxmind.geoip2.DatabaseReader.getCity(DatabaseReader.java:365) ~[graylog.jar:?]
at com.maxmind.geoip2.DatabaseReader.city(DatabaseReader.java:348) ~[graylog.jar:?]
at org.graylog.plugins.map.geoip.MaxMindIPLocationDatabaseAdapter.maxMindCity(MaxMindIPLocationDatabaseAdapter.java:39) ~[graylog.jar:?]
at org.graylog.plugins.map.geoip.MaxmindDataAdapter.doGet(MaxmindDataAdapter.java:186) ~[graylog.jar:?]
at org.graylog2.plugin.lookup.LookupDataAdapter.get(LookupDataAdapter.java:143) ~[graylog.jar:?]
at org.graylog2.lookup.LookupTable.lambda$lookup$0(LookupTable.java:73) ~[graylog.jar:?]
at org.graylog2.lookup.caches.CaffeineLookupCache.lambda$get$2(CaffeineLookupCache.java:161) ~[graylog.jar:?]
at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$2(LocalCache.java:167) ~[graylog.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2688) ~[graylog.jar:?]
at java.base/java.util.concurrent.ConcurrentHashMap.compute(Unknown Source) [?:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2686) [graylog.jar:?]
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2669) [graylog.jar:?]
at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112) [graylog.jar:?]
at com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62) [graylog.jar:?]
at org.graylog2.lookup.caches.CaffeineLookupCache.get(CaffeineLookupCache.java:182) [graylog.jar:?]
at org.graylog2.lookup.LookupTable.lookup(LookupTable.java:73) [graylog.jar:?]
at org.graylog2.lookup.LookupTableService$Function.lookup(LookupTableService.java:635) [graylog.jar:?]
at org.graylog2.inputs.extractors.LookupTableExtractor.run(LookupTableExtractor.java:66) [graylog.jar:?]
at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:226) [graylog.jar:?]
at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:79) [graylog.jar:?]
at org.graylog2.messageprocessors.MessageFilterChainProcessor.process(MessageFilterChainProcessor.java:105) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:167) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:137) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:107) [graylog.jar:?]
at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:52) [graylog.jar:?]
at org.graylog2.shared.buffers.PartitioningWorkHandler.onEvent(PartitioningWorkHandler.java:52) [graylog.jar:?]
at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167) [graylog.jar:?]
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122) [graylog.jar:?]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]

mmdb Sizes:
-rw-r--r-- 1 root root 237M Jul 26 00:38 GeoLite2-City-updated.mmdb
-rw-r--r-- 1 root root 50M Jul 25 07:28 GeoLite2-City.mmdb

json sizes:
-rw-r--r-- 1 root root 963M Jul 25 14:35 GeoLite2-City-Changes.json
-rw-r--r-- 1 root root 3.8G Jul 25 08:36 GeoLite2-City-Dump.json

Why is the database in some way corrupted that Graylog can't read but the command line can??