Missing files in on-demand mode #1214
Comments
I am struggling to think how this might be possible. My understanding how on demand works is that on the initial sync it creates a "remote artifact" which is essentially a download URL and some checksums. Once that artifact is needed for the first time, the file is downloaded and turned into a normal artifact that replaces the "remote artifact". At that point Pulp will never attempt to re-download the file again, so it should not be possible to get a 404. Are you absolutely certain that you are getting 404 on a file that was already downloaded (and not say, a package with the same name and version, but a different checksum i.e. a different file that has not yet been downloaded)? On a side note: The on demand implementation is entirely inherited from pulpcore. To my knowledge there are no pulp_deb plugin specific changes here. If we can find a clear reproducer for this issue, we will need to figure out if this is a pulpcore bug or some bad interaction with something on the plugin side. |
|
Hi, Thank you! log We have only one version of the repository. Artifact sha224 to identify file Open pulp content (kubectl exec) |
|
Just on the off chance that the file is present at That should return If the file is not corrupted, then it is there and Pulp should use the local file and not return 404 when it is accessed on the content app. I will try to get some pulpcore developer eyes on this issue. |
|
Thanks for your effort. Sha256sum returned the same hash. Should I escalate this issue to pulp-core? |
|
I'm not an expert for kubernetes nor the operator. It just sounds a bit weird to use file storage in such an environment. |
|
By checking the running processes I see that all services are running under the same user. The affected files are the same after redeploying the modules with the same user/group/owner permissions. |
|
@vogll Please open an issue in pulpcore. Make sure to link to this issue from the pulpcore one you open. As a rule I don't like punting problems to somebody else, but I just don't know enough about the on_demand implementation to help. While some weird interaction between core and plugin is always possible, the feature itself lives in core and I don't see how to make any progress without getting pulpcore developers involved. |
|
I did a new deployment and a new synchronization of Noble Ubuntu-updates. I found that all the same files were missing. That is, all the same file names as in the previous synchronization. Could it be a problem inside the deb (some regex) ? Here is list... |
|
What's up with the double slash (//) between amd64 and pool? |
This is created using a test script. Tests with a single slash have the same effect. |
Is this an immediate sync or an on_demand sync again? |
|
On_demand |
|
Seems to be that we found cause of the issue in the deb_module. By comparing the version construction with the core module, we found that the deb_module uses a different version format. As a result, some artifacts cannot be found. Below is a comparison of some failed modules: Comparsion for the package vim-common When querying for the relative path: It fails. Ubuntu-standart seems to be correct The issue may lie in how the core and deb modules construct the relative path version/filename. The different ways in which the version is represented between core and deb modules might be causing some artifacts to be unrecognized. |
|
I need to double check but my initial thought is that the deb_package should list the relative_path as actually synced from the upstream repo, while the publication artifact may by design diverge from this relative_path, because it uses the relative path format preferred/generated by the pulp_deb plugin. The affected cases you have identified appear to be packages with a version epoch that is not zero. This is a strong indication that the reason for the problem is related to a non-zero version epoch. This in turn suggests that this is some bug within the pulp_deb plugin after all (and not something to do with the pulpcore on_demand sync). This gives us a good idea where we should continue to investigate. Now I just need to find the time to do so. |
|
Note to self: This issue in pulpcore: pulp/pulpcore#6200 (Update once we are sure if this is core or plugin). |
|
Here is a workaround we used for this issue that might be helpful for debugging :
|
Version
"versions": [
{
"component": "core",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulpcore.app",
"domain_compatible": true
},
{
"component": "deb",
"version": "3.4.0",
"package": "pulp_deb",
"module": "pulp_deb.app",
"domain_compatible": false
},
{
"component": "ostree",
"version": "2.4.4",
"package": "pulp-ostree",
"module": "pulp_ostree.app",
"domain_compatible": true
},
{
"component": "certguard",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulp_certguard.app",
"domain_compatible": true
},
{
"component": "file",
"version": "3.69.0",
"package": "pulpcore",
"module": "pulp_file.app",
"domain_compatible": true
},
{
"component": "container",
"version": "2.22.0",
"package": "pulp-container",
"module": "pulp_container.app",
"domain_compatible": false
},
{
"component": "ansible",
"version": "0.23.1",
"package": "pulp-ansible",
"module": "pulp_ansible.app",
"domain_compatible": false
},
{
"component": "python",
"version": "3.12.5",
"package": "pulp-python",
"module": "pulp_python.app",
"domain_compatible": true
},
{
"component": "maven",
"version": "0.8.1",
"package": "pulp-maven",
"module": "pulp_maven.app",
"domain_compatible": false
},
{
"component": "rpm",
"version": "3.27.2",
"package": "pulp-rpm",
"module": "pulp_rpm.app",
"domain_compatible": true
}
],
Describe the bug
We have been using Pulp in on-demand mode for the Ubuntu archives (noble, noble-updates, noble-security) and only for the amd64 architecture. After several months of use, we observed that some artifacts previously used, reports 404 even though the files are present in /var/lib/pulp.
We haven't performed any resynchronizations other than the initial sync. For all repositories (noble, noble-updates, noble-security), we have created their own remote.
Pulp has been deployed in our Kubernetes cluster using the Pulp Operator. After the first occurence of the missing files we performed a redeploy.
Additional context
Here is the script I've used to test artifact presence:
Here is the script for the initial sync
The text was updated successfully, but these errors were encountered: