Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MCLEAN-124] Leverage Files.delete(Path) API to provide more accurate reason in case of failure #60

Merged
merged 6 commits into from
Nov 9, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,12 @@ under the License.
<artifactId>junit-jupiter-api</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
<version>4.11.0</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.codehaus.plexus</groupId>
<artifactId>plexus-testing</artifactId>
Expand Down
18 changes: 14 additions & 4 deletions src/main/java/org/apache/maven/plugins/clean/Cleaner.java
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,8 @@ private boolean isSymbolicLink(Path path) throws IOException {
* @throws IOException If a file/directory could not be deleted and <code>failOnError</code> is <code>true</code>.
*/
private int delete(File file, boolean failOnError, boolean retryOnError) throws IOException {
if (!file.delete()) {
IOException failure = delete(file);
if (failure != null) {
boolean deleted = false;

if (retryOnError) {
Expand All @@ -289,18 +290,18 @@ private int delete(File file, boolean failOnError, boolean retryOnError) throws
} catch (InterruptedException e) {
// ignore
}
deleted = file.delete() || !file.exists();
deleted = delete(file) == null || !file.exists();
}
} else {
deleted = !file.exists();
}

if (!deleted) {
if (failOnError) {
throw new IOException("Failed to delete " + file);
throw new IOException("Failed to delete " + file, failure);
} else {
if (log.isWarnEnabled()) {
log.warn("Failed to delete " + file);
log.warn("Failed to delete " + file, failure);
}
return 1;
}
Expand All @@ -310,6 +311,15 @@ private int delete(File file, boolean failOnError, boolean retryOnError) throws
return 0;
}

private static IOException delete(File file) {
try {
Files.delete(file.toPath());
} catch (IOException e) {
return e;
}
return null;
}

private static class Result {

private int failures;
Expand Down
173 changes: 173 additions & 0 deletions src/test/java/org/apache/maven/plugins/clean/CleanerTest.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.maven.plugins.clean;

import java.io.IOException;
import java.nio.file.AccessDeniedException;
import java.nio.file.DirectoryNotEmptyException;
import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.attribute.PosixFilePermission;
import java.nio.file.attribute.PosixFilePermissions;
import java.util.Set;

import org.apache.maven.plugin.logging.Log;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.ArgumentCaptor;
import org.mockito.InOrder;

import static java.nio.file.Files.createDirectory;
import static java.nio.file.Files.createFile;
import static java.nio.file.Files.exists;
import static java.nio.file.Files.getPosixFilePermissions;
import static java.nio.file.Files.setPosixFilePermissions;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assumptions.assumeTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.inOrder;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;

class CleanerTest {

private static final boolean POSIX_COMPLIANT =
FileSystems.getDefault().supportedFileAttributeViews().contains("posix");

private final Log log = mock();

@Test
void deleteSucceedsDeeply(@TempDir Path tempDir) throws Exception {
assumeTrue(POSIX_COMPLIANT);
final Path basedir = createDirectory(tempDir.resolve("target")).toRealPath();
Copy link
Contributor Author

@peterdemaeyer peterdemaeyer Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an attempt to fix the tests so that they also work on Windows.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might want to do this in a separate pr first

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I can't. This code doesn't exist outside of this PR, so I can't create a separate PR for it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do the tests work now on windows?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I disabled the tests on non-POSIX-compliant OSes using JUnit 5 Assumptions. I deemed it acceptable, because this is not a test for specific features and failures on all OSes, it's a test for Cleaner providing it with a variety of failures. It doesn't matter if these failure are exactly the same on all OSes, as long as there is at least one POSIX-compliant OS in the test suite, which is the case, it's enough to cover the feature.

final Path file = createFile(basedir.resolve("file"));
final Cleaner cleaner = new Cleaner(null, log, false, null, null);
cleaner.delete(basedir.toFile(), null, false, true, false);
assertFalse(exists(basedir));
assertFalse(exists(file));
}

@Test
void deleteFailsWithoutRetryWhenNoPermission(@TempDir Path tempDir) throws Exception {
assumeTrue(POSIX_COMPLIANT);
when(log.isWarnEnabled()).thenReturn(true);
final Path basedir = createDirectory(tempDir.resolve("target")).toRealPath();
createFile(basedir.resolve("file"));
final Set<PosixFilePermission> initialPermissions = getPosixFilePermissions(basedir);
final String rwxrwxr_x = PosixFilePermissions.toString(initialPermissions);
// Prevent directory listing, which will result in a DirectoryNotEmptyException.
final String rw_rw_r__ = rwxrwxr_x.replace('x', '-');
final Set<PosixFilePermission> permissions = PosixFilePermissions.fromString(rw_rw_r__);
setPosixFilePermissions(basedir, permissions);
try {
final Cleaner cleaner = new Cleaner(null, log, false, null, null);
final IOException exception =
assertThrows(IOException.class, () -> cleaner.delete(basedir.toFile(), null, false, true, false));
verify(log, never()).warn(any(CharSequence.class), any(Throwable.class));
assertEquals("Failed to delete " + basedir, exception.getMessage());
final DirectoryNotEmptyException cause =
assertInstanceOf(DirectoryNotEmptyException.class, exception.getCause());
assertEquals(basedir.toString(), cause.getMessage());
} finally {
// Allow the tempDir to be cleared by the @TempDir extension.
setPosixFilePermissions(basedir, initialPermissions);
}
}

@Test
void deleteFailsAfterRetryWhenNoPermission(@TempDir Path tempDir) throws Exception {
assumeTrue(POSIX_COMPLIANT);
final Path basedir = createDirectory(tempDir.resolve("target")).toRealPath();
createFile(basedir.resolve("file"));
final Set<PosixFilePermission> initialPermissions = getPosixFilePermissions(basedir);
final String rwxrwxr_x = PosixFilePermissions.toString(initialPermissions);
// Prevent directory listing, which will result in a DirectoryNotEmptyException.
final String rw_rw_r__ = rwxrwxr_x.replace('x', '-');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is replace needed here? Could this just be a string literal?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace is not strictly needed, it could be a string literal, but the replace reflects better the intention. I just want to remove the executable flag, and leave any other "rw" settings as they were. That's exactly what this replace expresses. If a make it a string literal, it is not clear to the reader that in fact I only mean to change the executable flag. The name of the variable rw_rw_r__ is merely suggestive, to reflect the sort of value the string can have.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels too clever by half. Let's just make it a string literal.

final Set<PosixFilePermission> permissions = PosixFilePermissions.fromString(rw_rw_r__);
setPosixFilePermissions(basedir, permissions);
try {
final Cleaner cleaner = new Cleaner(null, log, false, null, null);
final IOException exception =
assertThrows(IOException.class, () -> cleaner.delete(basedir.toFile(), null, false, true, true));
assertEquals("Failed to delete " + basedir, exception.getMessage());
final DirectoryNotEmptyException cause =
assertInstanceOf(DirectoryNotEmptyException.class, exception.getCause());
assertEquals(basedir.toString(), cause.getMessage());
} finally {
// Allow the tempDir to be cleared by the @TempDir extension.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be what I'm missing. The annotation doesn't work with the permissions you set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it does, maybe it doesn't. I didn't check. I deem it good practice to reset any modified persistent state so that I don't have to make assumptions about how "powerful" the @TempDir extension really is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked, the @TempDir framework is smart enough to fix the permissions to that the tmp dir can be deleted. So I removed all the try-catches.

setPosixFilePermissions(basedir, initialPermissions);
}
}

@Test
void deleteLogsWarningWithoutRetryWhenNoPermission(@TempDir Path tempDir) throws Exception {
assumeTrue(POSIX_COMPLIANT);
when(log.isWarnEnabled()).thenReturn(true);
final Path basedir = createDirectory(tempDir.resolve("target")).toRealPath();
final Path file = createFile(basedir.resolve("file"));
final Set<PosixFilePermission> initialPermissions = getPosixFilePermissions(basedir);
final String rwxrwxr_x = PosixFilePermissions.toString(initialPermissions);
final String r_xr_xr_x = rwxrwxr_x.replace('w', '-');
final Set<PosixFilePermission> permissions = PosixFilePermissions.fromString(r_xr_xr_x);
setPosixFilePermissions(basedir, permissions);
try {
final Cleaner cleaner = new Cleaner(null, log, false, null, null);
assertDoesNotThrow(() -> cleaner.delete(basedir.toFile(), null, false, false, false));
verify(log, times(2)).warn(any(CharSequence.class), any(Throwable.class));
InOrder inOrder = inOrder(log);
ArgumentCaptor<AccessDeniedException> cause1 = ArgumentCaptor.forClass(AccessDeniedException.class);
inOrder.verify(log).warn(eq("Failed to delete " + file), cause1.capture());
assertEquals(file.toString(), cause1.getValue().getMessage());
ArgumentCaptor<DirectoryNotEmptyException> cause2 =
ArgumentCaptor.forClass(DirectoryNotEmptyException.class);
inOrder.verify(log).warn(eq("Failed to delete " + basedir), cause2.capture());
assertEquals(basedir.toString(), cause2.getValue().getMessage());
} finally {
setPosixFilePermissions(basedir, initialPermissions);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need to reset this here? Feels like the object shouldn't be used again

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid resource exhaustion. It's good practice to reset persistent state which is modified in a test. The files I make in this test are persistent, and I rely on the @TempDir extension in the JUnit framework to delete them. If I leave permissions as they are, i.e. deliberately restricted so that files cannot be deleted, I imagine it could be that deleting the directory fails, resulting not per se in an exception but in lingering temporary directories. Over time, lingering temporary directories can cause "no space left on device" on test systems. By resetting the permissions, I reset the persistent state, allowing the JUnit framework to clear the temporary directory, avoiding resource exhaustion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something's off with that. Test resources including files should not be shared between test methods. Per Junit docs "The temporary directory will be shared by all tests in a class when the annotation is present on a static field or on a parameter of a @BeforeAll method. Otherwise — for example, when @tempdir is only used on instance fields or on parameters in test, @BeforeEach, or @AfterEach methods — each test will use its own temporary directory."

Copy link
Contributor Author

@peterdemaeyer peterdemaeyer Nov 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there is a misunderstanding. I agree that test resources including files should not be shared between test methods. That's exactly why I did not share them between tests. According to you quote of the JUnit docs, they fall in the category "@TempDir is used on parameters in tests", in which case "each test will use its own temporary directory", which is exactly what we want, no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I don't think these are what I would call "persistent'. Either way, let's start with the assumption you don't need this. Take out the finally block and see if everything still passes.

}
}

@Test
void deleteDoesNotLogAnythingWhenNoPermissionAndWarnDisabled(@TempDir Path tempDir) throws Exception {
assumeTrue(POSIX_COMPLIANT);
when(log.isWarnEnabled()).thenReturn(false);
final Path basedir = createDirectory(tempDir.resolve("target")).toRealPath();
createFile(basedir.resolve("file"));
final Set<PosixFilePermission> initialPermissions = getPosixFilePermissions(basedir);
final String rwxrwxr_x = PosixFilePermissions.toString(initialPermissions);
final String r_xr_xr_x = rwxrwxr_x.replace('w', '-');
final Set<PosixFilePermission> permissions = PosixFilePermissions.fromString(r_xr_xr_x);
setPosixFilePermissions(basedir, permissions);
try {
final Cleaner cleaner = new Cleaner(null, log, false, null, null);
assertDoesNotThrow(() -> cleaner.delete(basedir.toFile(), null, false, false, false));
verify(log, never()).warn(any(CharSequence.class), any(Throwable.class));
} finally {
setPosixFilePermissions(basedir, initialPermissions);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why reset?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my other answer.

}
}
}