Skip to content

macOS binaries: UV_FS_COPYFILE_FICLONE not supported #24521

Closed
Closed
macOS binaries: UV_FS_COPYFILE_FICLONE not supported#24521
Labels
confirmed-bugIssues and PRs for confirmed bugs.macosIssues and PRs related to the macOS platform.

Description

@cristianl
cristianl
opened on Nov 20, 2018

Node binaries from the official pkg distribution do not support copy-on-write (UV_FS_COPYFILE_FICLONE) on apfs-formatted volumes.

  • Version: v11.2.0
  • Platform: Darwin iMac.local 17.7.0 Darwin Kernel Version 17.7.0: Wed Oct 10 23:06:14 PDT 2018; root:xnu-4570.71.13~1/RELEASE_X86_64 x86_64
  • Subsystem: fs

Steps to reproduce

Generate a file (3MB here, size is not important) to clone later

mkdir ~/apfs-cow-test && cd ~/apfs-cow-test
head -c 3000000 </dev/urandom >origfile
node

In the node REPL, run:

const fs = require('fs')
fs.copyFile('origfile', 'newfile', fs.constants.UV_FS_COPYFILE_FICLONE_FORCE, err => { if (err) { console.log(err) }})

In the official node binary, this error is raised:

> { [Error: ENOSYS: function not implemented, copyfile 'origfile' -> 'newfile']
  errno: -78,
  code: 'ENOSYS',
  syscall: 'copyfile',
  path: 'origfile',
  dest: 'newfile' }

With a binary that supports this feature, the file is cloned normally. You can verify that the clone call works as intended by cloning a large file and checking the volume size on Disk Utility. It doesn't create a hardlink. Changes to the clone don't affect the original file.

Implications

This bug also affects users of nvm and n who install prebuilt binaries (the default behavior). brew users are not impacted as the prebuilt binary (bottle) is built with support for this flag.

Yarn is directly affected by this. UV_FS_COPYFILE_FICLONE falls back to normal copying, negating the performance and disk space benefits of cloning.

Activity

  1. bnoordhuis commented on Nov 20, 2018

    @bnoordhuis
    bnoordhuis
    on Nov 20, 2018
    Member

    With a binary that supports this feature

    I'm unclear on what you mean by that: another node binary, or another program entirely? If it's the former, where did you get it from?

  2. cristianl commented on Nov 20, 2018

    @cristianl
    cristianl
    on Nov 20, 2018
    Author

    Another node binary. 11.2.0, installed via brew install node. I'd like to try building from source but I'm afraid V8 would take some time.

  3. bnoordhuis
    added
    macosIssues and PRs related to the macOS platform.
    on Nov 20, 2018

  4. bnoordhuis commented on Nov 20, 2018

    @bnoordhuis
    bnoordhuis
    on Nov 20, 2018
    Member

    Okay, I understand what you mean now. It sounds like the release binaries were built on a system whose headers don't define COPYFILE_CLONE_FORCE. The ENOSYS comes from an #ifdef guard in libuv.

    @nodejs/releasers Can one of you confirm? I'm having a hard time tracking down the machine used to build them. The define should be in /usr/include/copyfile.h (that's where it is on my system anyway.)

  5. targos commented on Nov 20, 2018

    @targos
    targos
    on Nov 20, 2018
    Member

    @bnoordhuis version 11.2.0 was built on release-macstadium-macos10.11-x64-1.
    I do not have access to the machine so I can only give you a few lines from the build output that contain version numbers:

    21:12:59 Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
21:12:59 Apple LLVM version 8.0.0 (clang-800.0.42.1)
21:12:59 Target: x86_64-apple-darwin15.0.0

    Version 15.0.0 seems to be OS X El Capitan.

  6. rvagg commented on Nov 21, 2018

    @rvagg
    rvagg
    on Nov 21, 2018
    Member

    Yep, it's missing COPYFILE_CLONE_FORCE. Header says /* version 0.1 */ if that helps.

    Darwin release-macstadium-macos10.11-x64-1.nodejs.org 15.0.0 Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64 x86_64

  7. bnoordhuis commented on Nov 22, 2018

    @bnoordhuis
    bnoordhuis
    on Nov 22, 2018
    Member

    I had a look and I think it's unfixable short-term. COPYFILE_CLONE_FORCE was added in macos 10.12 and the release machine is 10.11.

    Libuv can't hard-code the value because copyfile() blissfully ignores flags it doesn't know about. That would break the API contract that states UV_FS_COPYFILE_FICLONE_FORCE fails with an error when the file system doesn't support cloning.

    Theoretically, libuv could call clonefileat/fclonefileat() directly instead of going through copyfile() but that's basically reimplementing copyfile(), and probably poorly.

    edit: Perhaps libuv can turn the compile-time guard into a runtime OS version check. Needs investigation.

  8. bnoordhuis
    added
    confirmed-bugIssues and PRs for confirmed bugs.
    on Nov 22, 2018

  9. rvagg commented on Nov 23, 2018

    @rvagg
    rvagg
    on Nov 23, 2018
    Member

    So, does this call for switching to 10.12+ for Node 12 release builds?

  10. bnoordhuis
    added a commit that references this issue on Nov 24, 2018

    darwin: use runtime check for file cloning

    256158e

  11. bnoordhuis commented on Nov 24, 2018

    @bnoordhuis
    bnoordhuis
    on Nov 24, 2018
    Member

    @rvagg I think we can work around this in libuv, see libuv/libuv#2092.

  12. bnoordhuis
    added a commit that references this issue on Nov 25, 2018

    darwin: use runtime check for file cloning

    77c30f2
  13. bnoordhuis
    closed this as completedin libuv/libuv#2092on Nov 25, 2018

  14. bnoordhuis commented on Nov 25, 2018

    @bnoordhuis
    bnoordhuis
    on Nov 25, 2018
    Member

    This should be fixed the next time libuv is upgraded, probably sometime next month.

  15. cjihrig
    added a commit that references this issue on Dec 19, 2018

    deps: upgrade to libuv 1.24.1

    728b155
  16. MylesBorins
    added a commit that references this issue on Dec 25, 2018

    deps: upgrade to libuv 1.24.1

    155d1d5
  17. refack
    added a commit that references this issue on Jan 14, 2019

    deps: upgrade to libuv 1.24.1

    035c2c7
  18. njlr
    added a commit that references this issue on Apr 5, 2019

    darwin: use runtime check for file cloning

    5bc38de
  19. BethGriggs
    added a commit that references this issue on Apr 17, 2019

    deps: upgrade to libuv 1.24.1

    07c5cda
  20. BethGriggs
    added a commit that references this issue on Apr 28, 2019

    deps: upgrade to libuv 1.24.1

    3c75964
  21. BethGriggs
    added a commit that references this issue on May 10, 2019

    deps: upgrade to libuv 1.24.1

    da25605
  22. MylesBorins
    added a commit that references this issue on May 16, 2019

    deps: upgrade to libuv 1.24.1

    4875e88
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    confirmed-bugIssues and PRs for confirmed bugs.macosIssues and PRs related to the macOS platform.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions