Pull f2fs update from Jaegeuk Kim:
"In this round, we've mainly focused on performance tuning and critical
bug fixes occurred in low-end devices. Sheng Yong introduced
lost_found feature to keep missing files during recovery instead of
thrashing them. We're preparing coming fsverity implementation. And,
we've got more features to communicate with users for better
performance. In low-end devices, some memory-related issues were
fixed, and subtle race condtions and corner cases were addressed as
well.
Enhancements:
- large nat bitmaps for more free node ids
- add three block allocation policies to pass down write hints given by user
- expose extension list to user and introduce hot file extension
- tune small devices seamlessly for low-end devices
- set readdir_ra by default
- give more resources under gc_urgent mode regarding to discard and cleaning
- introduce fsync_mode to enforce posix or not
- nowait aio support
- add lost_found feature to keep dangling inodes
- reserve bits for future fsverity feature
- add test_dummy_encryption for FBE
Bug fixes:
- don't use highmem for dentry pages
- align memory boundary for bitops
- truncate preallocated blocks in write errors
- guarantee i_times on fsync call
- clear CP_TRIMMED_FLAG correctly
- prevent node chain loop during recovery
- avoid data race between atomic write and background cleaning
- avoid unnecessary selinux violation warnings on resgid option
- GFP_NOFS to avoid deadlock in quota and read paths
- fix f2fs_skip_inode_update to allow i_size recovery
In addition to the above, there are several minor bug fixes and clean-ups"
Cherry-pick from origin/upstream-f2fs-stable-linux-4.4.y:
42bf67fc54 f2fs: remain written times to update inode during fsync
6cb5aa02bf f2fs: make assignment of t->dentry_bitmap more readable
a8d07f1f9c f2fs: truncate preallocated blocks in error case
86444d6006 f2fs: fix a wrong condition in f2fs_skip_inode_update
db2188a687 f2fs: reserve bits for fs-verity
ee2e74b3f0 f2fs: Add a segment type check in inplace write
0192e0a450 f2fs: no need to initialize zero value for GFP_F2FS_ZERO
49338842e9 f2fs: don't track new nat entry in nat set
d6a69d5e65 f2fs: clean up with F2FS_BLK_ALIGN
2c8834a7a2 f2fs: check blkaddr more accuratly before issue a bio
6ab573a9d9 f2fs: Set GF_NOFS in read_cache_page_gfp while doing f2fs_quota_read
7419dcb8be f2fs: introduce a new mount option test_dummy_encryption
9321e22c03 f2fs: introduce F2FS_FEATURE_LOST_FOUND feature
8a57196158 f2fs: release locks before return in f2fs_ioc_gc_range()
739ace131c f2fs: align memory boundary for bitops
4c55abe4f8 f2fs: remove unneeded set_cold_node()
30654507e0 f2fs: add nowait aio support
d909e94106 f2fs: wrap all options with f2fs_sb_info.mount_opt
5738be52b3 f2fs: Don't overwrite all types of node to keep node chain
0bdeb167c8 f2fs: introduce mount option for fsync mode
6bc490f0ee f2fs: fix to restore old mount option in ->remount_fs
0c9c3e0344 f2fs: wrap sb_rdonly with f2fs_readonly
6c6611223a f2fs: avoid selinux denial on CAP_SYS_RESOURCE
076a6f32fe f2fs: support hot file extension
58edcdbca6 f2fs: fix to avoid race in between atomic write and background GC
1e0aeb0af9 f2fs: do gc in greedy mode for whole range if gc_urgent mode is set
10b2d001d6 f2fs: issue discard aggressively in the gc_urgent mode
a5052f32b9 f2fs: set readdir_ra by default
1aa536a624 f2fs: add auto tuning for small devices
0ffdffc8f1 f2fs: add mount option for segment allocation policy
b798298912 f2fs: don't stop GC if GC is contended
766d232169 f2fs: expose extension_list sysfs entry
98b329de50 f2fs: fix to set KEEP_SIZE bit in f2fs_zero_range
4d409fa334 f2fs: introduce sb_lock to make encrypt pwsalt update exclusive
1f6bac14c1 f2fs: remove redundant initialization of pointer 'p'
946aefc754 f2fs: flush cp pack except cp pack 2 page at first
e5081a52ac f2fs: clean up f2fs_sb_has_xxx functions
a292477154 f2fs: remove redundant check of page type when submit bio
190e64a819 f2fs: fix to handle looped node chain during recovery
889d980876 f2fs: handle quota for orphan inodes
92b12bb1a2 f2fs: support passing down write hints to block layer with F2FS policy
22fa74c2b0 f2fs: support passing down write hints given by users to block layer
180900373e f2fs: fix to clear CP_TRIMMED_FLAG
0671fae134 f2fs: support large nat bitmap
eceb943d5d f2fs: fix to check extent cache in f2fs_drop_extent_tree
2e2a339c98 f2fs: restrict inline_xattr_size configuration
41dda11641 f2fs: fix heap mode to reset it back
39575737bb f2fs: fix potential corruption in area before F2FS_SUPER_OFFSET
7e0e7995ee fscrypt: fix build with pre-4.6 gcc versions
31d3279a4f fscrypt: fix up fscrypt_fname_encrypted_size() for internal use
82bec88856 fscrypt: define fscrypt_fname_alloc_buffer() to be for presented names
168a907828 fscrypt: calculate NUL-padding length in one place only
042ae9f4cf fscrypt: move fscrypt_symlink_data to fscrypt_private.h
f9550c24c2 fscrypt: remove fscrypt_fname_usr_to_disk()
7ac4756a24 f2fs: switch to fscrypt_get_symlink()
6b76f58e24 f2fs: switch to fscrypt ->symlink() helper functions
fd457d2c4e fscrypt: new helper function - fscrypt_get_symlink()
a1cdacb7ae fscrypt: new helper functions for ->symlink()
7f43602f4d fscrypt: trim down fscrypt.h includes
d9cadc11bd fscrypt: move fscrypt_is_dot_dotdot() to fs/crypto/fname.c
e6fe930580 fscrypt: move fscrypt_valid_enc_modes() to fscrypt_private.h
efefa434f4 fscrypt: move fscrypt_operations declaration to fscrypt_supp.h
7ed178bc8a fscrypt: split fscrypt_dummy_context_enabled() into supp/notsupp versions
3f16e09dad fscrypt: move fscrypt_ctx declaration to fscrypt_supp.h
8216a0b51a fscrypt: move fscrypt_info_cachep declaration to fscrypt_private.h
dfe0b3b1b6 fscrypt: move fscrypt_control_page() to supp/notsupp headers
3a2c791778 fscrypt: move fscrypt_has_encryption_key() to supp/notsupp headers
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
This is cherry-picked from upstrea-f2fs-stable-linux-4.4.y.
Changes include:
commit c7fd9e2b4a ("f2fs: hurry up to issue discard after io interruption")
commit 603dde3965 ("f2fs: fix to show correct discard_granularity in sysfs")
...
commit 565f0225f9 ("f2fs: factor out discard command info into discard_cmd_control")
commit c4cc29d19e ("f2fs: remove batched discard in f2fs_trim_fs")
Change-Id: Icd8a85ac0c19a8aa25cd2591a12b4e9b85bdf1c5
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
commit ec91538dccd44329ad83d3aae1aa6a8389b5c75f upstream.
This patch adds to set io_size_bits from mount option.
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Enable inline_data feature by default since it brings us better
performance and space utilization and now has already stable.
Add another option noinline_data to disable it during mount.
Suggested-by: Jaegeuk Kim <jaegeuk@kernel.org>
Suggested-by: Chao Yu <chao2.yu@samsung.com>
Signed-off-by: Wanpeng Li <wanpeng.li@linux.intel.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch adds a mount option 'extent_cache' in f2fs.
It is try to use a rb-tree based extent cache to cache more mapping information
with less memory if this option is set, otherwise we will use the original one
extent info cache.
Suggested-by: Changman Lee <cm224.lee@samsung.com>
Signed-off-by: Chao Yu <chao2.yu@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch introduces a batched trimming feature, which submits split discard
commands.
This is to avoid long latency due to huge trim commands.
If fstrim was triggered ranging from 0 to the end of device, we should lock
all the checkpoint-related mutexes, resulting in very long latency.
Reviewed-by: Chao Yu <chao2.yu@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch adds a mount option, norecovery, which is mostly same as
disable_roll_forward. The only difference is that norecovery should be activated
with read-only mount option.
This can be used when user wants to check whether f2fs is mountable or not
without any recovery process. (e.g., xfstests/200)
Reviewed-by: Chao Yu <chao2.yu@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
If a system wants to reduce the booting time as a top priority, now we can
use a mount option, -o fastboot.
With this option, f2fs conducts a little bit slow write_checkpoint, but
it can avoid the node page reads during the next mount time.
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch adds descriptions for the inline dir support in f2fs document.
Signed-off-by: Chao Yu <chao2.yu@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch changes the ipu_policy setting to use any combination of orthogonal policies.
Signed-off-by: Changman Lee <cm224.lee@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
If user wrote F2FS_IPU_FSYNC:4 in /sys/fs/f2fs/ipu_policy, f2fs_sync_file
only starts to try in-place-updates.
And, if the number of dirty pages is over /sys/fs/f2fs/min_fsync_blocks, it
keeps out-of-order manner. Otherwise, it triggers in-place-updates.
This may be used by storage showing very high random write performance.
For example, it can be used when,
Seq. writes (Data) + wait + Seq. writes (Node)
is pretty much slower than,
Rand. writes (Data)
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
This patch adds a mount option, nobarrier, in f2fs.
The assumption in here is that file system keeps the IO ordering, but
doesn't care about cache flushes inside the storages.
Reviewed-by: Chao Yu <chao2.yu@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
When large directory feathure is enable, We have one case which could cause
overflow in dir_buckets() as following:
special case: level + dir_level >= 32 and level < MAX_DIR_HASH_DEPTH / 2.
Here we define MAX_DIR_BUCKETS to limit the return value when the condition
could trigger potential overflow.
Changes from V1
o modify description of calculation in f2fs.txt suggested by Changman Lee.
Suggested-by: Changman Lee <cm224.lee@samsung.com>
Signed-off-by: Chao Yu <chao2.yu@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
Some storage devices show relatively high latencies to complete cache_flush
commands, even though their normal IO speed is prettry much high. In such
the case, it needs to merge cache_flush commands as much as possible to avoid
issuing them redundantly.
So, this patch introduces a mount option, "-o flush_merge", to mitigate such
the overhead.
If this option is enabled by user, F2FS merges the cache_flush commands and then
issues just one cache_flush on behalf of them. Once the single command is
finished, F2FS sends a completion signal to all the pending threads.
Note that, this option can be used under a workload consisting of very intensive
concurrent fsync calls, while the storage handles cache_flush commands slowly.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
It is more reasonable to determine the reclaiming rate of prefree segments
according to the volume size, which is set to 5% by default.
For example, if the volume is 128GB, the prefree segments are reclaimed
when the number reaches to 6.4GB.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This patch introduces ram_thresh, a sysfs entry, which controls the memory
footprint used by the free nid list and the nat cache.
Previously, the free nid list was controlled by MAX_FREE_NIDS, while the nat
cache was managed by NM_WOUT_THRESHOLD.
However, this approach cannot be applied dynamically according to the system.
So, this patch adds ram_thresh that users can specify the threshold, which is
in order of 1 / 1024.
For example, if the total ram size is 4GB and the value is set to 10 by default,
f2fs tries to control the number of free nids and nat caches not to consume over
10 * (4GB / 1024) = 10MB.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This patch adds an sysfs entry to control dir_level used by the large directory.
The description of this entry is:
dir_level This parameter controls the directory level to
support large directory. If a directory has a
number of files, it can reduce the file lookup
latency by increasing this dir_level value.
Otherwise, it needs to decrease this value to
reduce the space overhead. The default value is 0.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This patch introduces an i_dir_level field to support large directory.
Previously, f2fs maintains multi-level hash tables to find a dentry quickly
from a bunch of chiild dentries in a directory, and the hash tables consist of
the following tree structure as below.
In Documentation/filesystems/f2fs.txt,
----------------------
A : bucket
B : block
N : MAX_DIR_HASH_DEPTH
----------------------
level #0 | A(2B)
|
level #1 | A(2B) - A(2B)
|
level #2 | A(2B) - A(2B) - A(2B) - A(2B)
. | . . . .
level #N/2 | A(2B) - A(2B) - A(2B) - A(2B) - A(2B) - ... - A(2B)
. | . . . .
level #N | A(4B) - A(4B) - A(4B) - A(4B) - A(4B) - ... - A(4B)
But, if we can guess that a directory will handle a number of child files,
we don't need to traverse the tree from level #0 to #N all the time.
Since the lower level tables contain relatively small number of dentries,
the miss ratio of the target dentry is likely to be high.
In order to avoid that, we can configure the hash tables sparsely from level #0
like this.
level #0 | A(2B) - A(2B) - A(2B) - A(2B)
level #1 | A(2B) - A(2B) - A(2B) - A(2B) - A(2B) - ... - A(2B)
. | . . . .
level #N/2 | A(2B) - A(2B) - A(2B) - A(2B) - A(2B) - ... - A(2B)
. | . . . .
level #N | A(4B) - A(4B) - A(4B) - A(4B) - A(4B) - ... - A(4B)
With this structure, we can skip the ineffective tree searches in lower level
hash tables.
This patch adds just a facility for this by introducing i_dir_level in
f2fs_inode.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This patch adds missing some description of sysfs entries in
- Documentation/ABI/testing/sysfs-fs-f2fs
- Documentation/filesystems/f2fs.txt.
And it adds a maintained document entry of F2FS in MAINTAINERS.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This patch describes the inline_data support in f2fs document.
Signed-off-by: Huajun Li <huajun.li@intel.com>
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This patch introduces new sysfs entries for users to control the policy of
in-place-updates, namely IPU, in f2fs.
Sometimes f2fs suffers from performance degradation due to its out-of-place
update policy that produces many additional node block writes.
If the storage performance is very dependant on the amount of data writes
instead of IO patterns, we'd better drop this out-of-place update policy.
This patch suggests 5 polcies and their triggering conditions as follows.
[sysfs entry name = ipu_policy]
0: F2FS_IPU_FORCE all the time,
1: F2FS_IPU_SSR if SSR mode is activated,
2: F2FS_IPU_UTIL if FS utilization is over threashold,
3: F2FS_IPU_SSR_UTIL if SSR mode is activated and FS utilization is over
threashold,
4: F2FS_IPU_DISABLE disable IPU. (=default option)
[sysfs entry name = min_ipu_util]
This parameter controls the threshold to trigger in-place-updates.
The number indicates percentage of the filesystem utilization, and used by
F2FS_IPU_UTIL and F2FS_IPU_SSR_UTIL policies.
For more details, see need_inplace_update() in segment.h.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This patch adds a control method in sysfs to reclaim prefree segments.
Signed-off-by: Changman Lee <cm224.lee@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
Add sysfs entry gc_idle to control the gc policy. Where
gc_idle = 1 corresponds to selecting a cost benefit approach,
while gc_idle = 2 corresponds to selecting a greedy approach
to garbage collection. The selection is mutually exclusive one
approach will work at any point. If gc_idle = 0, then this
option is disabled.
Cc: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Namjae Jeon <namjae.jeon@samsung.com>
Signed-off-by: Pankaj Kumar <pankaj.km@samsung.com>
Reviewed-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
[Jaegeuk Kim: change the select_gc_type() flow slightly]
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
Add sysfs entries to control the timing parameters for
f2fs gc thread.
Various Sysfs options introduced are:
gc_min_sleep_time: Min Sleep time for GC in ms
gc_max_sleep_time: Max Sleep time for GC in ms
gc_no_gc_sleep_time: Default Sleep time for GC in ms
Cc: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Namjae Jeon <namjae.jeon@samsung.com>
Signed-off-by: Pankaj Kumar <pankaj.km@samsung.com>
Reviewed-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
[Jaegeuk Kim: fix an umount bug and some minor changes]
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This patch adds some description on fsck.f2fs and dump.f2fs which is
recently merged into f2fs-tools.
Signed-off-by: Changman Lee <cm224.lee@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
Add the f2fs_remount function call which will be used
during the filesystem remounting. This function
will help us to change the mount options specific to
f2fs.
Also modify the f2fs background_gc mount option, which
will allow the user to dynamically trun on/off the
garbage collection in f2fs based on the background_gc
value. If background_gc=on, Garbage collection will
be turned off & if background_gc=off, Garbage collection
will be truned on.
By default the garbage collection is on in f2fs.
Change Log:
v2: Incorporated the review comments by Gu Zheng.
Removing the restore part for VFS flags
Updating comments with proper flag conditions
Display GC background option as ON/OFF
Revised conditions to stop GC in case of remount
v1: Initial changes for adding remount_fs callback
support.
Cc: Gu Zheng <guz.fnst@cn.fujitsu.com>
Signed-off-by: Namjae Jeon <namjae.jeon@samsung.com>
Signed-off-by: Pankaj Kumar <pankaj.km@samsung.com>
Reviewed-by: Gu Zheng <guz.fnst@cn.fujitsu.com>
[Jaegeuk Kim: change /** with /* for the coding style]
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
o mkfs.f2fs supports no discard option.
o fixed volume label size in 512 bytes.
Signed-off-by: Changman Lee <cm224.lee@samsung.com>
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
document to reflect the layout generated by mkfs.f2fs .
Signed-off-by: Huajun Li <huajun.li.lee@gmail.com>
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
In f2fs_fs.h, one f2fs inode contains 923 data block pointers, while
f2fs documentation says it is 929. Fix this inconsistence.
Signed-off-by: Huajun Li <huajun.li.lee@gmail.com>
I moved the f2fs-tools.git into kernel.org.
And I added a new mailing list, linux-f2fs-devel@lists.sourceforge.net.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
This adds a document describing the mount options, proc entries, usage, and
design of Flash-Friendly File System, namely F2FS.
Signed-off-by: Jaegeuk Kim <jaegeuk.kim@samsung.com>
