Open Radar

 
Area: Apple Filesystem (APFS)

Problem: calling ftruncate does not hand space back to the operating system.

Steps to produce: pre-allocate space using fcntl, then ftruncate to a lower value. The file will still occupy the pre-allocated space, and there is (AFAICT) no way to shrink the file back down without deleting it and recreating it from scratch.

See/run the minimal C++ test case.

See also bug reports on this issue e.g. here: https://github.com/bitcoin/bitcoin/pull/17887

---attachment "apfsbug.cpp"---

#include <assert.h>
#include <cstdio>
#include <unistd.h>
#include <fcntl.h>

int main(int argc, char** argv) {
    FILE* file = fopen("myfile.dat", "wb");

    fstore_t fst = { F_ALLOCATECONTIG, F_PEOFPOSMODE, 0, 1024 * 1024 /* 1 MB */, 0 };
    if (fcntl(fileno(file), F_PREALLOCATE, &fst) == -1) {
        fst.fst_flags = F_ALLOCATEALL;
        assert(fcntl(fileno(file), F_PREALLOCATE, &fst) != -1);
    }
    ftruncate(fileno(file), 128); // truncate file down to 128 bytes
    fclose(file);
    // $ ls -la myfile.dat
    // -rw-r--r--  1 me  staff  128 Jan 13 13:43 myfile.dat
    // $ du -ch myfile.dat
    // 1.0M	myfile.dat
    // 1.0M	total
}

--- added info ---

I understand that you should be able to deallocate pre-allocated regions using the `F_PUNCHHOLE` operation. I tried this, but the results are exactly the same:
```C++
    fpunchhole_t fph = { 0, 0, 4096, 1024 * 1024 - 4096 };
    assert(-1 != fcntl(fileno(file), F_PUNCHHOLE, &fph));
    ftruncate(fileno(file), 128); // truncate file down to 128 bytes
    assert(-1 != fcntl(fileno(file), F_PUNCHHOLE, &fph));
    fclose(file);
```

---- Apple investigation results ----

Engineering has provided the following information regarding this issue:

Thanks for the note. Taking a look at your test program it seems that there are a few misunderstandings here about the APIs in question.

First: You mention that you cannot pre-allocate space using fcntl(F_PREALLOCATE) and get that space back using ftruncate(). That is true, but this is the expected behavior. Unlike posix_fallocate(), fcntl(F_PREALLOCATE) does not increase the logical size of the file, just the physical blocks allocated for them. In the man page for the operation, we say the following:

“ The position modes (fst_posmode) for the F_PREALLOCATE command indicate how to use the offset field.
     The modes are as follows:

           F_PEOFPOSMODE   Allocate from the physical end of file.

           F_VOLPOSMODE    Allocate from the volume offset.”

F_PEOFPOSMODE (as you are using it in the test) adds to the “physical end of file” - this is not the logical file size (meaning, it is not what fstat(2) would report in st_size - however, you’d notice this new size through an increase in st_blocks).

If you want to increase a file’s logical size after fcntl(F_PREALLOCATE), you must manually ftruncate() up to the size that you just allocated for the file. After doing this, an ftruncate() to a smaller size will deallocate that space.

Second: The reason fcntl(F_PUNCHHOLE) doesn’t work to do this either is that it operates on the logical file size. So, if you were to ftruncate() the logical size up to the size you preallocated, it should work just as well as another ftruncate().

We have a bug around here somewhere to eventually implement posix_fallocate(), which would work in the manner you’re expecting. So, if you’d like to track that feature, feel free to file a new bug for that.