Currently we initialize the heap free_lock and free list size in
ion_heap_init_deferred_free, which is only called when the
ION_HEAP_FLAG_DEFER_FREE heap flag is given. However, the lock and size
are used in the shrinker path as well as the deferred free path, and we
can register a shrinker *without* enabling deferred freeing. So, if a
heap provides a shrinker but *doesn't* set the DEFER_FREE flag we will
use these parameters uninitialized (resulting in a spinlock bug and
broken shrinker accounting).
Fix these problems by initializing the free list parameters directly in
ion_device_add_heap, which is always called no matter which heap
features are being used.
CRs-Fixed: 772761
Change-Id: Ib6466f940675040812b754690fd132ad95ec91d5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Set the private page flag on secure chunks to give other entities
a clue that the memory is special.
Change-Id: Ic0dedbad4b41a874a3f80dec75d0721d30307094
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
The cma secure heap allocates contiguous memory. This causes
the memory to be fragmented if several allocation and free
requests are made with buffers of various sizes. Introduce
the new ion flag(ION_FLAG_ALLOW_NON_CONTIG), which can be
passed to indicate a scatter gather list of smaller non
contiguous buffers can be allocated instead of a single
giant chunk.
Change-Id: I28b4ec8ee479e9cd565e1792e732137f0327bca3
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
[imaund@codeaurora.org: resolved context conflicts]
Signed-off-by: Ian Maund <imaund@codeaurora.org>
TZ expects the source and destination vm parameters as u32
integers. Define the parameters as u32 instead of u64.
Change-Id: I819427a1ebbb5823cf68acdbb5ea0c451ed313ea
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Add infrastructure to support a new system secure
heap, needed for secure use cases. The new heap
essentially allocates from the system heap, and
assigns the memory to the correct VM before
returning back to the client.
Change-Id: I2cd900e4399eecc3a9510139c4ebea0b713c39fc
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
We currently pass a NULL device when doing cache maintenance in
msm_ion_heap_buffer_zero. The dma ops that are used when a NULL device
is given rely on s->dma_address being the physical address of the
underlying memory. However, msm_ion_heap_buffer_zero can be called on
buffers that have previously been mapped by a different DMA mapper (like
the IOMMU DMA mapper) which might have set dma_address to something
other than the physical address of the underlying memory. This results
in us trying to do cache maintenance on some stuff that we shouldn't be.
Fix this by putting the physical address for the underlying memory back
into dma_address just before doing cache maintenance.
Change-Id: Ic5df328f5aeac09f7c9280ced887d2ba6098eb88
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
For certain secure use cases, the memory allocated using
ion needs to be re-assigned from one VM's MMU to another
VM's MMU. Call into the hypervisor to make this assingment.
Change-Id: I172f054e1b4385b8fcdaf7fdc2869034dbbd2db3
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
This reverts commit 49f5e2517691c768a884e7d1fb93306919081773.
This change results in some buffers not being zeroed in use
cases which did not show up in testing.
Change-Id: I659e7324bd21344444a0940b3403060254e8fba8
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Ian Maund <imaund@codeaurora.org>
Conflicts:
drivers/staging/android/ion/ion_page_pool.c
To support new usecases, HLOS will permit of sharing
of the CMA pool between the ion heaps. Going forward
we do not need to check if the heaps overlap.
Change-Id: I1dee72147fad994a1c88ea9dfcf7402ba98f3609
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Bugs in Ion clients are semi-common cause of out of memory
errors. Register with the show_mem notification framework
to let Ion dump out data for debugging. To avoid having to
take an excessive amount of locks, keep a running total of
allocated and potentially orphaned memory.
Change-Id: I52c31459d5bf975c67adb5f3fe17033c8e01d91b
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
All clients have been converted over to the new API
for securing. Instead of silently returning success when
secure fails, error out on a secure failure to allow for proper
error handling.
Change-Id: I57a5c1aef999cfc69a2a9cc1b3e28b2fa74fe3a6
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
If unsecure fails, this means the CPU cannot touch the
physical memory. This means if the memory is freed back to
the buddy allocator all accesses will trigger faults. If
unsecure fails, the only option is to leak the memory and
not free it.
Change-Id: I6a98761912598a9f8cdcc3914eca8e5e41147e55
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
The secure APIs are no longer called by external clients
and only called on alloc/free. The reference counting is now
useless so drop it.
Change-Id: Ide0fa6bb3f4183bbbe78d82b98166dc5942c6d3e
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
The usage hint is no longer used and is always 0. Remove
this as an argument to the lock/unlock APIs.
Change-Id: I7bafbe7823f26ba426a887e5c21f2db0852d3294
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
These APIs are deprecated as securing and unsecuring for secure
buffers is done on allocation and free respectively by the secure
heap layer. Clients don't have to call these explicitly.
Change-Id: If88cd1c47cba346446ebdcad494850b54ba954ab
Signed-off-by: Shalaj Jain <shalajj@codeaurora.org>
[mitchelh: dropped all client changes (since they're not in yet)]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
At one time, the zeroing algorithm used a mixture of
cached and uncached mappings which necessitated an extra
cache operation afterwards. The zeroing algorithm has been
simplified so the cache operation is no longer required if
the mapping used for zeroing matches the cache status of
the buffer (cached or not). Remove the cache operation and
use appropriate page protections instead.
Change-Id: I7d3596ed503811818782e77a0f3f7f63b1456d70
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
The scm library has added support for a new secure world
interface that is more aligned to the ARMv8 SMC calling
convention. Use the new API while maintaining backward
compatibility for the mem-protect call.
Change-Id: I3a4432f2eab4c52d0fb2e8aa5d5e9dd650fd05ef
Signed-off-by: Vikram Mulukutla <markivx@codeaurora.org>
The shrinker interface has now changed with separate count and scan
functions. Update this for the secure CMA shrinker.
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Many of the DT options for reserving memory are no longer used
at all. Simply the reading of bases/sizes from the DT and convert
to the new format.
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Users of dma on arm may call dma_*_writecombine and dma_*_nonconsistent.
Add these functions for arm64 as well.
Change-Id: I213f8314022969bcefbfcd8581725762c385fb18
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
This is a snapshot of the ION support as of msm-3.10 commit
acdce027751d5a7488b283f0ce3111f873a5816d (Merge "defconfig: arm64: Enable
ONESHOT_SYNC for msm8994")
In addition, comment out the shrinker code and skip-zeroing bits as they
aren't yet in the tree.
Change-Id: Id9e1e7fa4c35ce5a9f9348837f05f002258865cf
Signed-off-by: Kumar Gala <galak@codeaurora.org>
[mitchelh: dropped MSM changes to ion_chunk_heap, dropped MSM changes to
ion_carveout_heap]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The DMA framework currently zeros all buffers because it (righfully so)
assumes that drivers will soon need to pass the memory to a device.
Some devices/use case may not require zeroed memory and there can
be an increase in performance if we skip the zeroing. Add a DMA_ATTR
to allow skipping of DMA zeroing.
Change-Id: Id9ccab355554b3163d8e7eae1caa82460e171e34
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
[mitchelh: dropped changes to arm32]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
is_vmalloc_addr currently assumes that all vmalloc addresses
exist between VMALLOC_START and VMALLOC_END. This may not be
the case when interleaving vmalloc and lowmem. Update the
is_vmalloc_addr to properly check for this.
Correspondingly we need to ensure that VMALLOC_TOTAL accounts
for all the vmalloc regions when CONFIG_ENABLE_VMALLOC_SAVING
is enabled.
Change-Id: I5def3d6ae1a4de59ea36f095b8c73649a37b1f36
Signed-off-by: Susheel Khiani <skhiani@codeaurora.org>
Currently on 32 bit systems, virtual space above
PAGE_OFFSET is reserved for direct mapped lowmem
and part of virtual address space is reserved for
vmalloc. We want to optimize such as to have as
much direct mapped memory as possible since there is
penalty for mapping/unmapping highmem. Now, we may
have an image that is expected to have a lifetime of
the entire system and is reserved in physical region
that would be part of direct mapped lowmem. The
physical memory which is thus reserved is never used
by Linux. This means that even though the system is
not actually accessing the virtual memory
corresponding to the reserved physical memory, we
are still losing that portion of direct mapped lowmem
space.
So by allowing lowmem to be non contiguous we can
give this unused virtual address space of reserved
region back for use in vmalloc.
Change-Id: I980b3dfafac71884dcdcb8cd2e4a6363cde5746a
Signed-off-by: Susheel Khiani <skhiani@codeaurora.org>
During early init, all dma areas are remapped to PAGE_SIZE
granularity. Since full pmd regions are cleared to be
remapped into PAGE_SIZE, ensure that address range is pmd
size aligned while not crossing memory boundaries.
This would ensure that even if address region is not pmd
aligned, its mapping would not be cleared but factored in to
PAGE_SIZE regions.
Change-Id: Iad4ad7fd6169cdc693d532821aba453465addb7c
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
Even though lowmem is accounted for in vmalloc space,
allocation comes only from the region bounded by
VMALLOC_START and VMALLOC_END. The kernel virtual area
can now allocate from any unmapped region starting
from PAGE_OFFSET.
Change-Id: I291b9eb443d3f7445fd979bd7b09e9241ff22ba3
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Susheel Khiani <skhiani@codeaurora.org>
There are places in kernel like the lowmemorykiller which
invokes show_mem_call_notifiers from an atomic context.
So move from a blocking notifier to atomic. At present
the notifier callbacks does not call sleeping functions,
but it should be made sure, it does not happen in future also.
Change-Id: I9668e67463ab8a6a60be55dbc86b88f45be8b041
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
It was found that a number of tasks were blocked in the reclaim path
(throttle_vm_writeout) for seconds, because of vmstat_diff not being
synced in time. Fix that by adding a new function
global_page_state_snapshot.
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
Change-Id: Iec167635ad724a55c27bdbd49eb8686e7857216c
Commit "mm: vmscan: fix the page state calculation in too_many_isolated"
fixed an issue where a number of tasks were blocked in reclaim path
for seconds, because of vmstat_diff not being synced in time.
A similar problem can happen in isolate_migratepages_block, where
similar calculation is performed. This patch fixes that.
Change-Id: Ie74f108ef770da688017b515fe37faea6f384589
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
At present any vmpressure value is scaled up if the pages are
reclaimed through direct reclaim. This can result in false
vmpressure values. Consider a case where a device is booted up
and most of the memory is occuppied by file pages. kswapd will
make sure that high watermark is maintained. Now when a sudden
huge allocation request comes in, the system will definitely
have to get into direct reclaims. The vmpressures can be very low,
but because of allocstall accounting logic even these low values
will be scaled to values nearing 100. This can result in
unnecessary LMK kills for example. So define a tunable threshold
for vmpressure above which the allocstalls will be accounted.
CRs-fixed: 893699
Change-Id: Idd7c6724264ac89f1f68f2e9d70a32390ffca3e5
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
There are couple of issues with swapcache usage when ZRAM is used
as swap device.
1) Kernel does a swap readahead which can be around 6 to 8 pages
depending on total ram, which is not required for zram since
accesses are fast.
2) Kernel delays the freeing up of swapcache expecting a later hit,
which again is useless in the case of zram.
3) This is not related to swapcache, but zram usage itself.
As mentioned in (2) kernel delays freeing of swapcache, but along with
that it delays zram compressed page free also. i.e. there can be 2 copies,
though one is compressed.
This patch addresses these issues using two new flags
QUEUE_FLAG_FAST and SWP_FAST, to indicate that accesses to the device
will be fast and cheap, and instructs the swap layer to free up
swap space agressively, and not to do read ahead.
Change-Id: I5d2d5176a5f9420300bb2f843f6ecbdb25ea80e4
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
The existing calculation of vmpressure takes into account only
the ratio of reclaimed to scanned pages, but not the time spent
or the difficulty in reclaiming those pages. For e.g. when there
are quite a number of file pages in the system, an allocation
request can be satisfied by reclaiming the file pages alone. If
such a reclaim is successful, the vmpressure value will remain low
irrespective of the time spent by the reclaim code to free up the
file pages. With a feature like lowmemorykiller, killing a task
can be faster than reclaiming the file pages alone. So if the
vmpressure values reflect the reclaim difficulty level, clients
can make a decision based on that, for e.g. to kill a task early.
This patch monitors the number of pages scanned in the direct
reclaim path and scales the vmpressure level according to that.
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
Change-Id: I6e643d29a9a1aa0814309253a8b690ad86ec0b13
Currently, vmpressure is tied to memcg and its events are
available only to userspace clients. This patch removes
the dependency on CONFIG_MEMCG and adds a mechanism for
in-kernel clients to subscribe for vmpressure events (in
fact raw vmpressure values are delivered instead of vmpressure
levels, to provide clients more flexibility to take actions
on custom pressure levels which are not currently defined
by vmpressure module).
Change-Id: I38010f166546e8d7f12f5f355b5dbfd6ba04d587
Signed-off-by: Vinayak Menon <vinmenon@codeaurora.org>
Add a new config item, CONFIG_FORCE_ALLOC_FROM_DMA_ZONE, which
can be used to optionally force certain allocators to always
return memory from ZONE_DMA.
This option helps ensure that clients who require ZONE_DMA
memory are always using ZONE_DMA memory.
Change-Id: Id2d36214307789f27aa775c2bef2dab5047c4ff0
Signed-off-by: Liam Mark <lmark@codeaurora.org>
Currently we have kmemleak_stack_scan enabled by default.
This can hog the cpu with pre-emption disabled for a long
time starving other tasks.
Make this optional at compile time, since if required
we can always write to sysfs entry and enable this option.
Change-Id: Ie30447861c942337c7ff25ac269b6025a527e8eb
Signed-off-by: Vignesh Radhakrishnan <vigneshr@codeaurora.org>
Signed-off-by: Prasad Sodagudi <psodagud@codeaurora.org>
Currently we have kmemleak_stack_scan enabled by default.
This can hog the cpu with pre-emption disabled for a long
time starving other tasks.
Make this optional at compile time, since if required
we can always write to sysfs entry and enable this option.
Change-Id: Ie30447861c942337c7ff25ac269b6025a527e8eb
Signed-off-by: Vignesh Radhakrishnan <vigneshr@codeaurora.org>
Signed-off-by: Prasad Sodagudi <psodagud@codeaurora.org>
When CONFIG_PAGE_POISONING is enabled, the pages are poisoned
after setting free page in KASan Shadow memory and KASan reports
the read after free warning. The same thing happens in the allocation
path. So change the order of calling KASan_alloc/free API so that
pages poisoning happens when the pages are in alloc status in KASan
shadow memory.
following is the KASan report for reference.
==================================================================
BUG: KASan: use after free in memset+0x24/0x44 at addr ffffffc000000000
Write of size 4096 by task swapper/0
page:ffffffbac5000000 count:0 mapcount:0 mapping: (null) index:0x0
flags: 0x0()
page dumped because: kasan: bad access detected
CPU: 0 PID: 0 Comm: swapper Not tainted 3.18.0-g5a4a5d5-07242-g6938a8b-dirty #1
Hardware name: Qualcomm Technologies, Inc. MSM 8996 v2 + PMI8994 MTP (DT)
Call trace:
[<ffffffc000089ea4>] dump_backtrace+0x0/0x1c4
[<ffffffc00008a078>] show_stack+0x10/0x1c
[<ffffffc0010ecfd8>] dump_stack+0x74/0xc8
[<ffffffc00020faec>] kasan_report_error+0x2b0/0x408
[<ffffffc00020fd20>] kasan_report+0x34/0x40
[<ffffffc00020f138>] __asan_storeN+0x15c/0x168
[<ffffffc00020f374>] memset+0x20/0x44
[<ffffffc0002086e0>] kernel_map_pages+0x238/0x2a8
[<ffffffc0001ba738>] free_pages_prepare+0x21c/0x25c
[<ffffffc0001bc7e4>] __free_pages_ok+0x20/0xf0
[<ffffffc0001bd3bc>] __free_pages+0x34/0x44
[<ffffffc0001bd5d8>] __free_pages_bootmem+0xf4/0x110
[<ffffffc001ca9050>] free_all_bootmem+0x160/0x1f4
[<ffffffc001c97b30>] mem_init+0x70/0x1ec
[<ffffffc001c909f8>] start_kernel+0x2b8/0x4e4
[<ffffffc001c987dc>] kasan_early_init+0x154/0x160
Change-Id: Idbd3dc629be57ed55a383b069a735ae3ee7b9f05
Signed-off-by: Se Wang (Patrick) Oh <sewango@codeaurora.org>
For some use cases, it is not known beforehand, how much
removed (carve-out) region size must be reserved. Hence
the reserved region size might need to be adjusted to
support varying use cases. In such cases maintaining
different device tree configurations to support varying
carve-out region size is difficult.
Introduce an optional device tree property, to
reserved-memory, "no-map-fixup" which works in tandem with
"removed-dma-pool" compatibility that tries to shrink and
adjust the removed area on very first successful allocation.
At end of which it returns the additional (unused) pages
from the region back to the system.
Point to note is this that this adjustment is done on very
first allocation and thereafter the region size is big
enough only to support maximum of first allocation request
size. This fixup is attempted only once upon first
allocation and never after that. Clients can allocate and
free from this region as any other dma region.
As the description suggests this type of region is specific
to certain special needs and is not to be used for common
use cases.
Change-Id: I31f49d6bd957814bc2ef3a94910425b820ccc739
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
Using memset generates unaligned access exception for
device type memory on armv8, hence use memset_io for
ioremap region.
Change-Id: I26c82d4bed20f1c163953680aa200c95842d3f21
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
removed_region->base should be phy_addr_t type as it
directly holds physical memory address. Fix it.
Change-Id: I80d49d209cf0f319b7a468697387d23e6bcb1b98
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
Commit 54329ac (common: dma-mapping: introduce common remapping functions)
factored out common code for remapping arrays of pages. The code before
the refactor relied on setting area->pages with the array of mapped
pages for easy access later. The refactor dropped this, breaking
parts of the ARM DMA API. Fix this by setting the page array in the same
place.
Change-Id: Ie4d085132f350db29eb2aca67156c25b5e842903
Reported-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Large allocations can result in the dma_common_contiguous_remap
call not being able to succeed because it can't find enough
contiguous memory to setup the mapping.
Make dma_common_contiguous_remap more robust by using vmalloc
as a fallback.
Change-Id: I12ca710b4c24f4ef24bc33a0d1d4922196fb7492
Signed-off-by: Liam Mark <lmark@codeaurora.org>
Align size first and then find required number of bits and
order out of it.
Change-Id: I9b12fb45e5c1ff79e24fe7584cd23923b1a88c87
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
The current DMA coherent pool assumes that there is a kernel
mapping at all times for hte entire pool. This may not be
what we want for the entire times. Add the dma_removed ops to
support this use case.
Change-Id: Ie4f1e9bdf57b79699fa8fa7e7a6087e6d88ebbfa
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Change the logic which determines the initial readahead window size
such that for small requests (one page) the initial window size
will be x4 the size of the original request, regardless of the
VM_MAX_READAHEAD value. This prevents a rapid ramp-up
that could be caused due to increasing VM_MAX_READAHEAD.
Change-Id: I93d59c515d7e6c6d62348790980ff7bd4f434997
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
Memory watermarks were sometimes preventing CMA allocations
in low memory.
Change-Id: I550ec987cbd6bc6dadd72b4a764df20cd0758479
Signed-off-by: Liam Mark <lmark@codeaurora.org>
CMA allocations rely on being able to migrate pages out
quickly to fulfill the allocations. Most use cases for
movable allocations meet this requirement. File system
allocations may take an unaccpetably long time to
migrate, which creates delays from CMA. Prevent CMA
pages from ending up on the per-cpu lists to avoid
code paths grabbing CMA pages on the fast path. CMA
pages can still be allocated as a fallback under tight
memory pressure.
CRs-Fixed: 452508
Change-Id: I79a28f697275a2a1870caabae53c8ea345b4b47d
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
