Content Updater

[f-galland]

Description

A context is outdated when its offset and last_offset are different. For example, in the case below.

[
  {
    "_index": "wazuh-content",
    "_id": "vd_1.0.0",
    "_source": {
      "vd_4.8.0": {
        "offset": 75019,
        "last_offset": 85729
      }
    }
  },
]

In this case, the Content Manager downloads the changes in chunks. The total number of chunks to be fetched can be expressed as chunks = (last_offset - offset) / MAX_SIZE, where MAX_SIZE=1000. This is the upper limit of changes the CTI API returns in a single request. Following the example, the Content Manager would download 10 chunks of 1000 changes and 1 chunk of 710 changes, for a total of 11 requests to the CTI API and 10710 changes, that need to be processed and indexed.

The Content Updater module uses the CTI client module to fetch the changes. It then processes the data and transforms it into CUD (create, update, delete) operations to the appropriate index, using the Index Management module. When the update is complete, it generates a command for the Command Manager, using the client. The Content Updater is the orchestrator of the update process, delegating the fetching and indexing operations to other modules.

---
title: Content Manager - Content update
---
sequenceDiagram
    ContentUpdater->>IndexClient: getContextInformation()
    IndexClient-)ContentUpdater: contextInfo
    loop while last_offset > offset
        ContentUpdater->>CTIclient: getContextChanges()
        CTIclient-)ContentUpdater: changes
        ContentUpdater-->>IndexPatcher: changes
    end
    ContentUpdater-->>CommandManagerClient: postCommand()

Loading

Functional requirements

• The Content Updater checks if any context needs and update.
• The Content Updater updates a context up to the last offset.
• The Content Updater generates a command on a successful context update.

Implementation restrictions

• The Content Updater module relies on other modules for most of the operations: CTIClient, CommandManagerClient, IndexPatcher, ...
• During a context update, its offset value is updated on each iteration. When the update ends, offset must be equal to last_offset.

Plan

• Read contexts metadata from the internal index. Filter by offset != last_offset.
• Orchestrate the context update process.
• Generate a command on a succesful update.