Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement the solidity compilation cache #6129

Open
wants to merge 53 commits into
base: v-next
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
94a0a3c
wip: implement the compilation job and artifacts caches
galargh Jan 9, 2025
8635878
wip: try moving the artifacts higher
galargh Jan 10, 2025
6e89b99
chore: add version parameter to the cache constructor
galargh Jan 28, 2025
1a755bc
chore: remove the setFiles/getFiles methods from the cache interface
galargh Jan 28, 2025
a622510
chore: remove the artifacts cache
galargh Jan 28, 2025
7a20c12
chore: remove compiler output cache from run compilation job
galargh Jan 28, 2025
7c4f198
chore: add saving to cache to the build function
galargh Jan 28, 2025
100579e
feat: add cache cleanup function
galargh Jan 28, 2025
0787fcc
chore: clean cache only after all the writes are complete
galargh Jan 28, 2025
ed3328a
chore: remove unused has method from cache
galargh Jan 28, 2025
cd0adef
chore: move cache value type declaration to the constructor
galargh Jan 28, 2025
73ba8e0
chore: remove unused function from build system implementation
galargh Jan 28, 2025
b224e1a
chore: simplify successful result check
galargh Jan 28, 2025
41bddca
feat: make getBuildId and getSolcInput on CompilationJob async
galargh Jan 28, 2025
96096b2
chore: add helper functions for getting part of solc inputs
galargh Jan 28, 2025
0f4974a
feat: make hash generation asynchronous
galargh Jan 28, 2025
9b71988
feat: use hashed sources in the build id hash input
galargh Jan 28, 2025
b48e0b9
feat: cache resolved file content hashes
galargh Jan 28, 2025
f724079
fix: include sources in the compilation job id
galargh Jan 28, 2025
0d6e346
chore: put the source content hash cache on compilation job
galargh Jan 28, 2025
7ebad19
test: make build system tests easier to change
galargh Jan 29, 2025
ded19e2
chore: simplify the compiler cache namespace
galargh Jan 29, 2025
6386662
fix: make compilation job build id deterministic again
galargh Jan 29, 2025
9a32f70
fix: set sane defaults for cache options
galargh Jan 29, 2025
ff5bfd6
chore: fix lint in utils
galargh Jan 29, 2025
b28e802
chore: revert the changes to hardhat dependencies
galargh Jan 29, 2025
8660ad0
test: fix the hash creation tests after algorithm change
galargh Jan 29, 2025
b13cae7
chore: revert changes to the getSolcInput signature
galargh Jan 29, 2025
0593851
chore: remove unnecessary loops and recomputations
galargh Jan 29, 2025
1179386
test: implement new fs utils tests
galargh Jan 29, 2025
0a52ab7
fix: the build id should include solc config
galargh Jan 29, 2025
27d6a2d
chore: add a utility clear cache function to the compilation job
galargh Jan 29, 2025
e55db65
test: implement compilation job build id tests
galargh Jan 29, 2025
0b59516
chore: make object cache more suitable for testing
galargh Jan 29, 2025
b8eff76
test: implement object cache tests
galargh Jan 29, 2025
d14b385
chore: fix linting of the fs tests
galargh Jan 29, 2025
8f8a033
test: implement more build system tests
galargh Jan 29, 2025
416e4d0
Merge remote-tracking branch 'origin/v-next' into build-system-cache
galargh Jan 29, 2025
135d975
chore: fix tests after merging v-next
galargh Jan 29, 2025
c2de298
fix: provide implicitly required context for the hook manager
galargh Jan 29, 2025
35b8e96
chore: revert the addition of emit artifacts options
galargh Jan 29, 2025
7647d54
chore: remove async from a method that doesn't need it
galargh Jan 29, 2025
5ce2a3d
chore: remove forgotten describe.only
galargh Jan 29, 2025
c14cf21
chore: turn resolved files into classes
galargh Jan 30, 2025
222a6c8
chore: replace static source content hash with per resolved file one
galargh Jan 30, 2025
b1377d3
chore: move resolved file interface implementations out of types
galargh Jan 30, 2025
b1d6cad
test: add a 1ms sleep before cleaning cache
galargh Jan 30, 2025
0dbc016
chore: fix lint
galargh Jan 30, 2025
d8eeaa6
Merge branch 'v-next' into build-system-cache
galargh Jan 30, 2025
bb1d79f
Merge remote-tracking branch 'origin/v-next' into build-system-cache
galargh Feb 3, 2025
927b8cc
Merge remote-tracking branch 'origin/v-next' into build-system-cache
galargh Feb 5, 2025
28fe32e
chore: address feedback fromt the review
galargh Feb 5, 2025
1da1e5c
test: fix cache tests in which the access time order matters
galargh Feb 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions v-next/hardhat-utils/src/crypto.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import { createHash } from "node:crypto";

import { keccak256 as keccak256Impl } from "ethereum-cryptography/keccak";

/**
Expand All @@ -17,17 +15,21 @@ export async function keccak256(bytes: Uint8Array): Promise<Uint8Array> {
*
* This function is primarily intended for generating unique identifiers from
* a given input string.
* It uses the MD5 hash algorithm, which is not cryptographically secure, but
* It uses the SHA-1 hash algorithm, which is not cryptographically secure, but
* is sufficient for this use case as long as the input is not generated by an
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this LGTM

* attacker.
*
* Note: The exact algorithm used (MD5) is not crucial for the function's
* Note: The exact algorithm used (SHA-1) is not crucial for the function's
* purpose of generating unique identifiers, and could be replaced if needed.
*
* @param data The input string to be hashed.
* @returns The MD5 hash of the input string, represented as a
* @returns The SHA-1 hash of the input string, represented as a
* hexadecimal string.
*/
export function createNonCryptographicHashId(data: string): string {
return createHash("md5").update(data).digest("hex");
export async function createNonCryptographicHashId(
data: string,
): Promise<string> {
const message = new TextEncoder().encode(data);
const buffer = await crypto.subtle.digest("SHA-1", message);
return Buffer.from(buffer).toString("hex");
}
44 changes: 44 additions & 0 deletions v-next/hardhat-utils/src/fs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -407,6 +407,50 @@ export async function getChangeTime(absolutePath: string): Promise<Date> {
}
}

/**
* Retrieves the last access time of a file or directory's properties.
*
* @param absolutePath The absolute path to the file or directory.
* @returns The time of the last access as a Date object.
* @throws FileNotFoundError if the path does not exist.
* @throws FileSystemAccessError for any other error.
*/
export async function getAccessTime(absolutePath: string): Promise<Date> {
try {
const stats = await fsPromises.stat(absolutePath);
return stats.atime;
} catch (e) {
ensureError<NodeJS.ErrnoException>(e);
if (e.code === "ENOENT") {
throw new FileNotFoundError(absolutePath, e);
}

throw new FileSystemAccessError(e.message, e);
}
}
Comment on lines +418 to +430
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Context

Here, I add two new helper functions, getAccessTime and getSize which return last access time and size of a file respectively.

They are needed to implement the clean part of the caching component which removes cache entries based on file size and "age".


/**
* Retrieves the size of a file.
*
* @param absolutePath The absolute path to the file.
* @returns The size of the file in bytes.
* @throws FileNotFoundError if the path does not exist.
* @throws FileSystemAccessError for any other error.
*/
export async function getFileSize(absolutePath: string): Promise<number> {
try {
const stats = await fsPromises.stat(absolutePath);
return stats.size;
} catch (e) {
ensureError<NodeJS.ErrnoException>(e);
if (e.code === "ENOENT") {
throw new FileNotFoundError(absolutePath, e);
}

throw new FileSystemAccessError(e.message, e);
}
}

/**
* Checks if a file or directory exists.
*
Expand Down
6 changes: 3 additions & 3 deletions v-next/hardhat-utils/test/crypto.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ describe("crypto", () => {
});

describe("createNonCryptographicHashId", () => {
it("Should create a non-cryptographic hash-based identifier", () => {
it("Should create a non-cryptographic hash-based identifier", async () => {
assert.equal(
createNonCryptographicHashId("hello"),
"5d41402abc4b2a76b9719d911017c592",
await createNonCryptographicHashId("hello"),
"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d",
);
});
});
Expand Down
61 changes: 61 additions & 0 deletions v-next/hardhat-utils/test/fs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ import {
writeJsonFile,
writeUtf8File,
readBinaryFile,
getAccessTime,
getFileSize,
} from "../src/fs.js";

import { useTmpDir } from "./helpers/fs.js";
Expand Down Expand Up @@ -735,6 +737,65 @@ describe("File system utils", () => {
});
});

describe("getAccessTime", () => {
it("Should return the access time of a file", async () => {
const filePath = path.join(getTmpDir(), "file.txt");
await createFile(filePath);

const stats = await fsPromises.stat(filePath);

assert.equal(
stats.atime.getTime(),
(await getAccessTime(filePath)).getTime(),
);
});

it("Should throw FileNotFoundError if the file doesn't exist", async () => {
const filePath = path.join(getTmpDir(), "not-exists.txt");

await assert.rejects(getAccessTime(filePath), {
name: "FileNotFoundError",
message: `File ${filePath} not found`,
});
});

it("Should throw FileSystemAccessError if a different error is thrown", async () => {
const invalidPath = "\0";

await assert.rejects(getAccessTime(invalidPath), {
name: "FileSystemAccessError",
});
});
});

describe("getSize", () => {
it("Should return the size of a file", async () => {
const filePath = path.join(getTmpDir(), "file.txt");
await createFile(filePath);

const stats = await fsPromises.stat(filePath);

assert.equal(stats.size, await getFileSize(filePath));
});

it("Should throw FileNotFoundError if the file doesn't exist", async () => {
const filePath = path.join(getTmpDir(), "not-exists.txt");

await assert.rejects(getFileSize(filePath), {
name: "FileNotFoundError",
message: `File ${filePath} not found`,
});
});

it("Should throw FileSystemAccessError if a different error is thrown", async () => {
const invalidPath = "\0";

await assert.rejects(getFileSize(invalidPath), {
name: "FileSystemAccessError",
});
});
});

describe("exists", () => {
it("Should return true if the file exists", async () => {
const filePath = path.join(getTmpDir(), "file.txt");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,9 @@ declare module "@ignored/hardhat-vnext/types/artifacts" {
}`;
}

export function getBuildInfo(
export async function getBuildInfo(
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Context

It is necessary to change the signatures of getBuildInfo and getBuildInfoOutput because they both use getBuildId which turned asynchronous as part of the getBuildId optimisations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(these comments are really helpful)

compilationJob: CompilationJob,
): SolidityBuildInfo {
): Promise<SolidityBuildInfo> {
const publicSourceNameMap = Object.fromEntries(
[...compilationJob.dependencyGraph.getRoots().entries()].map(
([publicSourceName, root]) => [publicSourceName, root.sourceName],
Expand All @@ -115,7 +115,7 @@ export function getBuildInfo(

const buildInfo: Required<BuildInfo> = {
_format: "hh3-sol-build-info-1",
id: compilationJob.getBuildId(),
id: await compilationJob.getBuildId(),
solcVersion: compilationJob.solcConfig.version,
solcLongVersion: compilationJob.solcLongVersion,
publicSourceNameMap,
Expand All @@ -125,13 +125,13 @@ export function getBuildInfo(
return buildInfo;
}

export function getBuildInfoOutput(
export async function getBuildInfoOutput(
compilationJob: CompilationJob,
compilerOutput: CompilerOutput,
): SolidityBuildInfoOutput {
): Promise<SolidityBuildInfoOutput> {
const buildInfoOutput: Required<SolidityBuildInfoOutput> = {
_format: "hh3-sol-build-info-output-1",
id: compilationJob.getBuildId(),
id: await compilationJob.getBuildId(),
output: compilerOutput,
};

Expand Down
Loading
Loading