-
-
Notifications
You must be signed in to change notification settings - Fork 35.1k
Virtual File System for Node.js #61478
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mcollina
wants to merge
116
commits into
nodejs:main
Choose a base branch
from
mcollina:vfs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+18,945
−63
Open
Changes from all commits
Commits
Show all changes
116 commits
Select commit
Hold shift + click to select a range
cc08a06
fs: add virtual file system support
mcollina 7eabec2
vfs: add Windows path compatibility
mcollina c363709
sea: support VFS in embedderRequire
mcollina 8b1aa8f
test: add tmpdir.refresh() to SEA VFS test
mcollina 9a4e1b6
test: update SEA VFS test for new buildSEA API
mcollina 43adc6b
vfs: add provider-based architecture and node:vfs module
mcollina 4fe1158
vfs: remove backward compat methods, use standard fs API
mcollina 83f4fe7
vfs: address review comments
mcollina 402a81c
vfs: fix lint errors
mcollina 0f3e763
vfs: fix lint errors
mcollina 54af709
vfs: remove public createSEA()
mcollina 29c0418
vfs: address review comments
mcollina 91bfe25
doc: address review comments on VFS documentation
mcollina 3c5211d
doc: clarify virtualCwd behavior in Worker threads
mcollina 37300a1
vfs: add RealFSProvider for mounting real directories
mcollina a7b5862
tools: add VFS types to doc type-parser
mcollina 3fac286
doc: use REPLACEME for version placeholders in vfs.md
mcollina 86cb0a6
doc: add security warnings and symlink documentation to vfs.md
mcollina 9e8479b
vfs: address code review feedback from @jasnell
mcollina fab4453
vfs: add overlay mode for selective file interception
mcollina c035802
vfs: add tests and fix appendFile, add readonly checks
mcollina f46855c
fs: remove createVirtual, use node:vfs instead
mcollina 1ee25ca
vfs: add watch and watchFile support
mcollina 897b989
vfs: improve test coverage for watch implementation
mcollina 2266ac1
vfs: remove unused utility functions from module_hooks
mcollina 0585518
vfs: add test for symlink target creation after symlink
mcollina d5cf2b6
doc: add worker thread limitations to VFS documentation
mcollina 334f017
doc: clarify VFS accepts same types as fs module
mcollina 81d1395
vfs: remove unused entries.js, add error tests
mcollina bade36a
vfs: remove unused exports from fd.js and stats.js
mcollina da96b9f
vfs: remove unused VirtualFD methods
mcollina 65f4e6f
vfs: remove more unused VirtualFD code
mcollina 549d9d7
vfs: address Aviv's review comments
mcollina 799b773
test: improve VFS code coverage
mcollina af3aabe
Update lib/internal/vfs/module_hooks.js
mcollina b03e338
Update lib/internal/vfs/module_hooks.js
mcollina 9b37b32
vfs: address aduh95 review comments
mcollina 42a0813
vfs: remove SEAProvider export and hasSeaAssets
mcollina 33b9e9c
vfs: remove addFile and addDirectory methods
mcollina e4f943e
doc: alphabetize VirtualFileSystem members in vfs.md
mcollina 62067f4
Update single-executable-applications.md
mcollina c88b5ad
sea: remove getVfs from public API
mcollina f6ea36b
doc: explain symlink behavior in overlay mode
mcollina 63e2519
doc: document overlay mode behavior for fs operations
mcollina 51dca9d
vfs: add Symbol.dispose support for automatic unmount
mcollina d818dd1
vfs: add missing JSDoc @returns for mount()
mcollina 5f36f6e
vfs: only initialize SEA VFS when assets exist
mcollina 87804ee
vfs: use path.posix methods instead of custom implementations
mcollina f39028e
vfs: address review feedback for SEA docs and getSeaVfs
mcollina 8d61b19
vfs: refactor SEA VFS with initSeaVfs/getSeaVfs pattern
mcollina 1267775
vfs: address test review feedback
mcollina fc13205
vfs: replace custom path helpers with standard path module
mcollina 187ca4e
vfs: address remaining review feedback from avivkeller
mcollina 5044cfd
test: expand VFS provider and promises test coverage
mcollina a7600be
fixup! test: expand VFS provider and promises test coverage
mcollina 7cbc649
doc: address vfs documentation review feedback
mcollina a3ec3dd
test: add Windows-specific VFS mount path tests
mcollina a440080
vfs: address code review feedback
mcollina 206c7fe
vfs: add process vfs-mount and vfs-unmount events
mcollina 29818ec
vfs: address code review feedback
mcollina b80ea83
doc: add case sensitivity documentation for VFS overlay mode
mcollina 40bbe99
sea: use wrapModuleLoad for VFS module loading
mcollina 2115076
doc: add VFS limitations and cross-references
mcollina c79c99a
vfs: fix path normalization on Windows
mcollina ef1ead1
sea,vfs: address review feedback from Joyee
mcollina d4c4d26
test: use relative require in SEA VFS test
mcollina a893956
sea,vfs: fix C++ formatting and markdown link ordering
mcollina ff93075
doc: address Joyee's documentation review feedback
mcollina d79511a
sea,vfs: address Joyee's code review feedback
mcollina 8923b94
test: add buffer independence and SEA VFS coverage tests
mcollina cd4e861
sea: fix C++ compilation errors in Initialize
mcollina 7b9734a
vfs: replace fs patching with toggleable toRealPath for module loading
mcollina 0b279c4
vfs: extend toggleable wrappers to readFileSync and internalModuleStat
mcollina 5703cbc
vfs: make package.json reader VFS-aware
mcollina 0ed9884
vfs: convert underscore-prefixed methods to private class fields
mcollina 1965689
test: add missing callback tests for VFS promises test file
mcollina 5ed245c
vfs: lazy-load esm/formats to fix shared library builds
mcollina 2f5c616
test: add VFS bare specifier ESM resolution test
mcollina f6a87eb
vfs: replace monkeypatching with self-contained module.registerHooks
mcollina 74e002d
test: add comprehensive VFS module hooks coverage tests
mcollina 53f57c7
vfs: address review feedback for module_hooks and file_system
mcollina 5b6e3d1
test: use descriptive mount paths in VFS ESM import tests
mcollina af8ed7f
test: add VFS internals coverage tests
mcollina 17c4acb
Apply suggestions from code review
mcollina 7ea0ad2
Apply suggestion from @aduh95
mcollina 3f61858
Apply suggestions from code review
mcollina da9e560
doc: address aduh95 review feedback on VFS docs
mcollina 17858bd
vfs: address aduh95 code review comments
mcollina ba5b1d3
vfs: restore isExperimentalSeaWarningNeeded as a function
mcollina ef8b1e9
vfs: revert non-VFS changes in cjs/loader.js and esm/resolve.js
mcollina 2c9ae60
vfs: restore extensionless file format detection and test
mcollina fcd05a7
vfs: remove internalModuleStat from provider base class
mcollina 79afe93
vfs: align lib/internal/modules/ with upstream main
mcollina d187d6e
vfs: restore isSea as function, align modules/ with upstream
mcollina a020fbb
vfs: restore sea.js to match upstream main
mcollina 77c3c9a
vfs: use wrapModuleLoad for SEA VFS entry point
mcollina 2285720
vfs: replace fs monkey-patching with toggleable loader wrappers
mcollina b64733f
vfs: address review feedback on path utilities and providers
mcollina 79f4d6e
vfs: fix Windows path handling and remove wrapper functions
mcollina 19a694f
vfs: fix lint warnings in router.js and test-vfs-chdir.js
mcollina cd7f836
vfs: document that useVfs does not support ESM entry points
mcollina 5496a39
vfs: add VFS-aware wrappers for package.json reader methods
mcollina bafbf9c
vfs: remove vfs-mount and vfs-unmount process events
mcollina 6a53f62
vfs: add require(esm) and ESM detection tests
mcollina 7ed1fac
vfs: reject useVfs with useSnapshot, useCodeCache, or mainFormat module
mcollina abef360
vfs: move fs interception from monkey-patching into lib/fs.js
mcollina 85685d0
vfs: remove duplicated module resolution logic from setup.js
mcollina 84c1693
vfs: intercept all supported fs methods when VFS is mounted
mcollina 219d7fe
vfs: increase test coverage and fix lint in destructuring test
mcollina 523ffd1
vfs: add remaining fs method interceptions and openAsBlob support
mcollina 54806bd
vfs: add remaining fs/promises VFS interceptions
mcollina f79ec2f
vfs: convert isVfsEnabled from property to method
mcollina 220c430
vfs: add ESM loader intercepts for legacyMainResolve and getFormatOfE…
mcollina b894e3f
vfs: fix ESM legacyMainResolve tests to use bare specifier imports
mcollina 8728a45
vfs: improve test coverage and add c8 ignore for SEA paths
mcollina 2b95b97
vfs: address review comments on cp-sync and test docs
mcollina File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -116,6 +116,7 @@ The configuration currently reads the following top-level fields: | |
| "disableExperimentalSEAWarning": true, // Default: false | ||
| "useSnapshot": false, // Default: false | ||
| "useCodeCache": true, // Default: false | ||
| "useVfs": true, // Default: false | ||
| "execArgv": ["--no-warnings", "--max-old-space-size=4096"], // Optional | ||
| "execArgvExtension": "env", // Default: "env", options: "none", "env", "cli" | ||
| "assets": { // Optional | ||
|
|
@@ -175,6 +176,84 @@ const raw = getRawAsset('a.jpg'); | |
| See documentation of the [`sea.getAsset()`][], [`sea.getAssetAsBlob()`][], | ||
| [`sea.getRawAsset()`][] and [`sea.getAssetKeys()`][] APIs for more information. | ||
|
|
||
| ### Virtual File System (VFS) for assets | ||
mcollina marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| <!-- YAML | ||
| added: REPLACEME | ||
| --> | ||
|
|
||
| > Stability: 1 - Experimental | ||
|
|
||
| In addition to using the `node:sea` API to access individual assets, you can use | ||
| the Virtual File System (VFS) to access bundled assets through standard `node:fs` | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: the |
||
| APIs. To enable VFS, set `"useVfs": true` in the SEA configuration. When | ||
| enabled, the VFS is automatically initialized and mounted at `/sea`. All | ||
| assets defined in the SEA configuration are accessible through this virtual | ||
| path. | ||
|
|
||
| ```cjs | ||
| const fs = require('node:fs'); | ||
|
|
||
| // Assets are automatically available at /sea when running as SEA | ||
| const rawConfig = fs.readFileSync('/sea/config.json', 'utf8'); | ||
| const data = fs.readFileSync('/sea/data/file.txt'); | ||
|
|
||
| const config = JSON.parse(rawConfig); | ||
|
|
||
| // Directory operations work too | ||
| const files = fs.readdirSync('/sea/assets'); | ||
|
|
||
| // Check if a bundled file exists | ||
| if (fs.existsSync('/sea/optional.json')) { | ||
| // ... | ||
| } | ||
| ``` | ||
|
|
||
| The VFS supports all common `node:fs` operations for reading files and directories. | ||
| Since the SEA VFS is read-only, write operations are not supported. See the | ||
| [VFS documentation][] for the full list of supported and unsupported operations. | ||
|
|
||
| #### Loading modules from VFS in SEA | ||
|
|
||
| When `useVfs` is enabled, `require()` in the injected main script uses the | ||
| registered module hooks to load modules from the virtual file system. This | ||
| supports both absolute VFS paths and relative requires (e.g., | ||
| `require('./helper.js')` from a VFS module), as well as `node_modules` package | ||
| lookups. | ||
|
|
||
| ```cjs | ||
| // Require bundled modules using relative paths | ||
| const myModule = require('./lib/mymodule.js'); | ||
| const utils = require('./utils/helpers.js'); | ||
|
|
||
| // Absolute VFS paths also work | ||
| const config = require('/sea/config.json'); | ||
| ``` | ||
|
|
||
| When `useVfs` is enabled, `__filename` and `__dirname` in the injected main | ||
| script reflect VFS paths (e.g., `/sea/main.js`) instead of | ||
| [`process.execPath`][]. | ||
|
|
||
joyeecheung marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| #### ESM limitations | ||
|
|
||
| The `useVfs` option does not currently support ESM entry points. Using | ||
| `"useVfs": true` together with `"mainFormat": "module"` is not supported. | ||
| The main script must use CommonJS (`require()`) when VFS is enabled. | ||
|
|
||
| #### Code caching limitations | ||
|
|
||
| The `useCodeCache` option in the SEA configuration does not currently apply to | ||
| modules loaded from the VFS. This is a current limitation due to incomplete | ||
| implementation, not a technical impossibility. Consider bundling the application | ||
| to enable code caching and do not rely on module loading in VFS. | ||
|
|
||
| #### Native addon limitations | ||
|
|
||
| Native addons (`.node` files) cannot be loaded directly from the VFS because | ||
| `process.dlopen()` requires files on the real file system. To use native | ||
| addons in a SEA with VFS, write the asset to a temporary file first. See | ||
| [Using native addons in the injected main script][] for an example. | ||
|
|
||
| ### Startup snapshot support | ||
|
|
||
| The `useSnapshot` field can be used to enable startup snapshot support. In this | ||
|
|
@@ -647,6 +726,8 @@ to help us document them. | |
| [Generating single executable preparation blobs]: #1-generating-single-executable-preparation-blobs | ||
| [Mach-O]: https://en.wikipedia.org/wiki/Mach-O | ||
| [PE]: https://en.wikipedia.org/wiki/Portable_Executable | ||
| [Using native addons in the injected main script]: #using-native-addons-in-the-injected-main-script | ||
| [VFS documentation]: vfs.md | ||
| [Windows SDK]: https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/ | ||
| [`process.execPath`]: process.md#processexecpath | ||
| [`require()`]: modules.md#requireid | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -2393,6 +2393,98 @@ test('mocks a counting function', (t) => { | |||||||||||||
| }); | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ### `mock.fs([options])` | ||||||||||||||
|
|
||||||||||||||
| <!-- YAML | ||||||||||||||
| added: REPLACEME | ||||||||||||||
| --> | ||||||||||||||
|
|
||||||||||||||
| > Stability: 1.0 - Early development | ||||||||||||||
|
|
||||||||||||||
| * `options` {Object} Optional configuration options for the mock file system. | ||||||||||||||
| The following properties are supported: | ||||||||||||||
| * `prefix` {string} The mount point prefix for the virtual file system. | ||||||||||||||
| **Default:** `'/mock'`. | ||||||||||||||
| * `files` {Object} An optional object where keys are file paths (relative to | ||||||||||||||
| the VFS root) and values are the file contents. Contents can be strings, | ||||||||||||||
| Buffers, or functions that return strings/Buffers. | ||||||||||||||
| * Returns: {MockFSContext} An object that can be used to manage the mock file | ||||||||||||||
| system. | ||||||||||||||
|
|
||||||||||||||
| This function creates a mock file system using the [Virtual File System (VFS)][]. | ||||||||||||||
| The mock file system is automatically cleaned up when the test completes. | ||||||||||||||
|
|
||||||||||||||
| ## Class: `MockFSContext` | ||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
|
|
||||||||||||||
| <!-- YAML | ||||||||||||||
| added: REPLACEME | ||||||||||||||
| --> | ||||||||||||||
|
|
||||||||||||||
| The `MockFSContext` object is returned by `mock.fs()` and provides the | ||||||||||||||
| following methods and properties: | ||||||||||||||
|
|
||||||||||||||
| * `vfs` {VirtualFileSystem} The underlying VFS instance. | ||||||||||||||
| * `prefix` {string} The mount prefix. | ||||||||||||||
| * `addFile(path, content)` Adds a file to the mock file system. | ||||||||||||||
| * `addDirectory(path[, populate])` Adds a directory to the mock file system. | ||||||||||||||
| * `existsSync(path)` Checks if a path exists (path is relative to prefix). | ||||||||||||||
| * `restore()` Manually restores the file system to its original state. | ||||||||||||||
|
|
||||||||||||||
| The following example demonstrates how to create a mock file system for testing: | ||||||||||||||
|
|
||||||||||||||
| ```js | ||||||||||||||
| const { test } = require('node:test'); | ||||||||||||||
| const assert = require('node:assert'); | ||||||||||||||
| const fs = require('node:fs'); | ||||||||||||||
|
|
||||||||||||||
| test('reads configuration from mock file', (t) => { | ||||||||||||||
| const mockFs = t.mock.fs({ | ||||||||||||||
| prefix: '/app', | ||||||||||||||
| files: { | ||||||||||||||
| '/config.json': JSON.stringify({ debug: true }), | ||||||||||||||
| '/data/users.txt': 'user1\nuser2\nuser3', | ||||||||||||||
| }, | ||||||||||||||
| }); | ||||||||||||||
|
|
||||||||||||||
| // Files are accessible via standard fs APIs | ||||||||||||||
| const config = JSON.parse(fs.readFileSync('/app/config.json', 'utf8')); | ||||||||||||||
| assert.strictEqual(config.debug, true); | ||||||||||||||
|
|
||||||||||||||
| // Check file existence | ||||||||||||||
| assert.strictEqual(fs.existsSync('/app/config.json'), true); | ||||||||||||||
| assert.strictEqual(fs.existsSync('/app/missing.txt'), false); | ||||||||||||||
|
|
||||||||||||||
| // Use mockFs.existsSync for paths relative to prefix | ||||||||||||||
| assert.strictEqual(mockFs.existsSync('/config.json'), true); | ||||||||||||||
| }); | ||||||||||||||
|
|
||||||||||||||
| test('supports dynamic file content', (t) => { | ||||||||||||||
| let counter = 0; | ||||||||||||||
| const mockFs = t.mock.fs({ prefix: '/dynamic' }); | ||||||||||||||
|
|
||||||||||||||
| mockFs.addFile('/counter.txt', () => { | ||||||||||||||
| counter++; | ||||||||||||||
| return String(counter); | ||||||||||||||
| }); | ||||||||||||||
|
|
||||||||||||||
| // Each read calls the function | ||||||||||||||
| assert.strictEqual(fs.readFileSync('/dynamic/counter.txt', 'utf8'), '1'); | ||||||||||||||
| assert.strictEqual(fs.readFileSync('/dynamic/counter.txt', 'utf8'), '2'); | ||||||||||||||
| }); | ||||||||||||||
|
|
||||||||||||||
| test('supports require from mock files', (t) => { | ||||||||||||||
| t.mock.fs({ | ||||||||||||||
| prefix: '/modules', | ||||||||||||||
| files: { | ||||||||||||||
| '/math.js': 'module.exports = { add: (a, b) => a + b };', | ||||||||||||||
| }, | ||||||||||||||
| }); | ||||||||||||||
|
|
||||||||||||||
| const math = require('/modules/math.js'); | ||||||||||||||
| assert.strictEqual(math.add(2, 3), 5); | ||||||||||||||
| }); | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ### `mock.getter(object, methodName[, implementation][, options])` | ||||||||||||||
|
|
||||||||||||||
| <!-- YAML | ||||||||||||||
|
|
@@ -4259,3 +4351,4 @@ Can be used to abort test subtasks when the test has been aborted. | |||||||||||||
| [suite options]: #suitename-options-fn | ||||||||||||||
| [test reporters]: #test-reporters | ||||||||||||||
| [test runner execution model]: #test-runner-execution-model | ||||||||||||||
| [virtual file system (vfs)]: vfs.md | ||||||||||||||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: add the parallel mjs examples