Unhandled context faults should be fatal unless explicitly requested
otherwise by the client. Implement the DOMAIN_ATTR_NON_FATAL_FAULTS
attribute for this purpose.
Change-Id: I414fcc6000a6c47b3cbf77b1098c8b7895cbb20d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
During development, any IOMMU faults most likely need to be debugged
immediately. Allowing the system to continue to run has a tendency to
mask the original problem, which makes debugging more difficult. Some
drivers might want to make all faults fatal by default, unless
explicitly requested otherwise by a client. Add a domain attribute for
this.
Change-Id: I454b143ea3cae1f3de36e0888afd15936aa63ce1
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The clock and regulator APIs can fail, but we're ignoring their return
values. This results in strange code paths where we misuse the
API (e.g. we enable clocks after a failed prepare, which makes the clock
driver yell). Fix this by checking all return values and propagating
the errors up to clients as needed.
CRs-Fixed: 924600
Change-Id: Ib6cb3afac1f5472f3a5abb5c251765344fe1a7a9
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The device config probe messages are for informational purposes only.
Move them to debug level to avoid spamming the boot log unnecessarily.
CRs-Fixed: 918584
Change-Id: If9085e2165f26b66ff94b258894f01471be82a5b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we only take one sample per buffer size per API when doing
profiling. This results in high run-to-run variance in the results.
Use more iterations to help smooth this out.
Change-Id: I3779007a2f69ef79b573285b2422554f42dda99f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Secure domains do different amounts of work than non-secure domains, so
we need to be able to profile them separately. Add another debugfs
file, "secure_profiling", that does the same thing as the "profiling"
file.
Change-Id: I6968036bfdcc63449f27336771b3956a2b7754f6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
secure_buffer.h makes use of the scatterlist structure, but doesn't
include the header file that defines it. Until now we've been getting
lucky that all users of secure_buffer.h have been including the
scatterlist header file themselves, but this might not always be the
case. Add the missing include.
Change-Id: If7598c26a1ef3551132c6eb62ac9bd6d447654d0
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The hypervisor assign call has substantial overhead, so calling it every
time we need to allocate some page table memory is currently killing
performance. Fix this by hanging on to secure page table memory when
it's freed, keeping it in a pool of pre-assigned memory for future
allocations.
Change-Id: Iefa45e6bc23e841d168be4f17c1f669f2e932a3b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We've been slowly littering arm-smmu.c with forward declarations all
over the place, moving them around when they're not high enough in the
file, etc. Just move them all to the top of the file once and for all.
Change-Id: I8f2f075546f287db17f1b6e3b08390296cb9b9ff
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Simultaneous maps and/or unmaps will lead to a race condition,
where if the assign list has multiple entries and one call is
in the middle of an assign operation for one entry, and the other
call removes the next element from the list and queues itself for
the assign operation. When the first call returns from assign, and
the list is empty, it will return back to the caller without
completing the assign on all the ptes allocated for that mapping.
This results in a page fault when the client is actually accessing
the buffer.
Protect the assign and unassign lists with a mutex to ensure that
the operation is serialized and the control does not return back
to the client before all the ptes have been correctly assigned.
Change-Id: Idaf894671fee9fc9c8d339bfa17344b68dd0ad77
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Currently there's a check to make sure that a mapping isn't being
installed on top of an existing leaf mapping. However, mapping on top
of an existing table mapping is silently ignored. Make the check more
strict by testing for any valid page table entry (table, block, or page
descriptor) and by making any overlap fatal.
Change-Id: Ibdffbfdb2b44cf9da5044c2c1828a32979a39760
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
When we brought back hardware ATOS support [94bc89042be5cb:
"iommu/arm-smmu: Resurrect hardware iova-to-phys"] we added some macros
back that were never deleted in the first place. Clean up the redundant
macros.
Change-Id: I0129313fa4132f45fe5805e9ed3477ef7819e70e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The MICRO_MMU_CTRL register is used for halt control, but it also
contains some important configuration bits. Currently, when we request
a halt, we're wiping out all of the other bits in the register, losing
any previous configuration. Fix this by reading the value and OR'ing in
the bits we need.
Change-Id: Iafdc5b2a344916dce9bac7f958a153dbee042ca8
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some SMMUs may not have a dedicated gdsc for it.
Allow for the enable code to prepare clocks and vote
for bus requests even if there is no gdsc provided.
Change-Id: Iaedfb73d8ae7f12895472893468e5c32d2bf2462
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
When using the client fault handler, set resume to terminate the
faulting transaction similar to the way of the default fault
handler.
Its possible that if the client fault handler doesn't "fully"
handle the fault, there would be a stream of page faults on the
same address since we would keep on retrying the same
transaction.
Change-Id: Id7aa439fd90aa5b63364c8ac270593330c55d672
Signed-off-by: Shrenuj Bansal <shrenujb@codeaurora.org>
Allow the creation of privileged mode mappings, for stage 1 only.
Change-Id: Ie91168012b61c0774d840a35b3ed2f18b834873f
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
It's not uncommon for us to take a flood of context faults when
something terrible happens. This usually brings down the system due to
excessive logging, which makes it difficult to debug the actual root
cause of the issue. Fix this by rate-limiting the output from the
context fault handler.
Change-Id: I1fe83bbbc7cc7aa2cf6303c6c7d2bf5733f63557
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
DMA allocations with no kernel mapping are likely
protected as well. Stale highmem mappings in such cases
with cache-able attributes may lead to speculative fetch
,as highmem mappings are not cleared out.
Hence flush out unused highmem mappings explicitly when
allocation request with no kernel mapping is requested.
Change-Id: Ic1de633c6364eaa1b6d5b0932f2cfe17d64d920e
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
The DMA_ATTR_NO_KERNEL_MAPPING is used to make sure that CMA
pages have no kernel mapping. Add support to make sure that
highmem pages have no mapping.
Change-Id: Ife76df126ecfedf0dba81a35e0de8a1787355b3d
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
Currently, there are use cases where not having any kernel
mapping is required; if the CMA memory needs to be used as
a pool which can have both cached and uncached mappings we
need to remove the mapping to avoid the multiple mapping
problem. Extend the dma APIs to use the DMA_ATTR_NO_KERNEL_MAPPING
with CMA. This doesn't end up saving any virtual address space
but the mapping will still not be present.
Change-Id: I64d21250abbe615c43e2b5b1272ee2b6d106705a
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
With the dma attribute 'SKIP_ZEROING', dma allocation skips memset and
also cache flush operations. This leaves dirty lines in the caches which
when evicted later would cause issue to the allocated buffer. For some
clients this may be okay but other clients need to do the actual sync.
Switch the code to only skip __dma_clear_buffer if both
DMA_ATTR_SKIP_ZEROING and DMA_ATTR_SKIP_CPU_SYNC are specified. If only
DMA_ATTR_SKIP_ZEROING is specified, just skip the zeroing but still do the
cache operations.
Change-Id: I81dd6bbeed7ec7f1dce28cd8d0aba7c8900ec550
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
The DMA framework currently zeros all buffers because it (righfully so)
assumes that drivers will soon need to pass the memory to a device.
Some devices/use case may not require zeroed memory and there can
be an increase in performance if we skip the zeroing. Add a DMA_ATTR
to allow skipping of DMA zeroing.
Change-Id: Ib78a19cb341c3c441f91d5b004c6375c80c10413
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Some SMMU operations (like register access and ATOS) require downstream
buses to be enabled. We've been getting lucky for a long time since
other clients have been voting for the buses in question, but recently
we've seen some unclocked accesses due to our missing vote. Add support
for bus scale voting to the driver. The voting parameters will be
provided later in the per-target device tree SMMU configuration.
Change-Id: I6b31c887ad817a6628f4234622c73dee046d6442
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The subdirectories under "attach" currently get a UUID appended to their
name to ensure that there won't be any collisions when multiple domains
are attached to the same device. This is only needed for dynamic
domains since those are the only types of domains that can be attached
to the same device concurrently. Only do the uniquification for dynamic
domains since the resulting directory name is a bit messy with the UUID.
Change-Id: Ia86b192e5b9d6a63132228fa86549a2977ec74a4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There's some non-deterministic latency incurred by enabling clocks and
regulators. Clocks and regulators are left on for atomic domains, which
makes them a good choice for profiling the page table management code.
Use an atomic domain for map/unmap profiling to make things more
deterministic.
Change-Id: Ib9fa48699c9d6009dd3f3b322002e2fb0f2ff99c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The iommu domain must be detached before the pagetable is freed,
if any TLB maintenance is needed it should be done by
iommu_detach_device().
Change-Id: I119d0e127e122de1263c80c73d40ff6edf16c875
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
The pagetable memory for a domain is freed by free_io_pgtable_ops(), which
is only called from arm_smmu_destroy_domain_context(). This function also
cleans up the hardware's context bank state and it is called during
iommu_device_detach(). Because dynamic domains don't have context bank
state, they don't call arm_smmu_destroy_domain_context() and thus their
pagetable memory was leaked.
Fix this by instead calling free_io_pgtable_ops() during
iommu_domain_destroy(). This ensures that the pagetable is freed for all
domains.
Change-Id: Id349fdc7df673cd4513468f23dcc95a9281d2657
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
We cannot call unmap directly if the map_sg fails partially as the tlb
invalidate functions need to enable/prepare clocks, which require
non-atomic context. Let map_sg return the failure and handle this when
we are out of the atomic context.
Change-Id: I6401c1e281850aeda27e32524cae34324045f762
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
It can be useful to read and modify SMMU settings interactively. Add a
debugfs interface for the recently added iommu_reg_read and
iommu_reg_write functions for this purpose.
Change-Id: If3c9bf1833dc950c7ca1e2ce1a373bcaa31ebcf0
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Implement the register reader and writer functions so that clients can
peek and poke their SMMU's registers.
Change-Id: I9dac59a16b42fec6b2a7639603289911f88b067d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It might be useful for IOMMU clients to peek and poke at their IOMMU's
registers, but knowing how to access those registers is really the job
of the IOMMU driver (it might need to enable specific clocks and
regulators, for example). Provide an API to read and write IOMMU
registers that can be implemented by the driver.
Change-Id: I5b2f19225f8bd258278780ff24b4ea96460857aa
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
For all but the last level of the page tables, we
need to call unmap with supported IOMMU pages sizes.
However currently to get the maximum page size that can be
unmapped, we are passing the original size parameter,
instead of the remaining size to be unmapped, and that can
end up unmapping more than what is intended.
Fix this by passing the remaining size parameter to
find the correct IOMMU page size.
Change-Id: Id4bd188ff6a3b4b7d34ba43ae6a61efb3c65b281
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Call the correct function to free the io_pgtable_ops when
there is an error.
Change-Id: Ib01041f874a203b18265b46b56b375c76c0ab211
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
iommu_trigger_fault now passes a flag value when triggering faults. Use
that value instead of hard-coding a translation fault.
Change-Id: Iac9b97cebef99e260641bde70c1ddfaedd0be502
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
As per the hardware spec, the SMMU must be halted during programming of
the implementation defined registers. Add the missing halt.
Change-Id: Id28b546e5b5b646c263d7431a32f3c95ed3f2042
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
iommu_trigger_fault now accepts a flags argument which can be handled by
the individual IOMMU drivers. Take the value written to the debugfs
file by the user and pass it as the flags value.
Change-Id: I803e6585aea35fbb0f6e03e4279587b36a7bad92
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
IOMMU drivers might want more control over the types of faults being
triggered with iommu_trigger_fault. Add a flags parameter that can be
used to provide more control over the types of faults that can be
triggered.
Change-Id: I2f21b383437430e957ab52070d3575e8cb3dee90
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
A client fault handler flag was recently introduced to indicate external
faults. Use it.
Change-Id: Ib9ea59befc3c8324de4fc94a0dbe4acafaa6f9b8
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some IOMMUs (like the ARM SMMU) can raise "external faults" in addition
to translation and permission faults. Add an IOMMU fault flag so that
this fault type can be propagated to client fault handlers.
Change-Id: Ia476ed6c727f5f01011747f30b3a74ebf0fc07f7
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Simultaneous assign or unassign on the same domain during map and unmap
operations manipulate the same pte_info lists and thus can ask hypervisor
to assign the same pte address twice which results in double mapping error.
The lists(pte_info_list and unassign_list) needs to be locked before
traversing it for making the assign/unassign calls.
Change-Id: Iebe3d703d80918af4ab18208f4b5a010c4c14791
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
When unmapping 2MB mappings, which are 2MB aligned, the smmu driver
is leaking the 3rd level page tables.
Fix this leak by updating __arm_lpae_free_pgtable so that it no
longer leaks leaf table entries.
To reproduce this leak simply map and unmap a non-block 2MB mapping
which is 2MB aligned.
Change-Id: Ibdbdb084ceb8d03ebe0a04e8777e3eb9419e9b87
Signed-off-by: Liam Mark <lmark@codeaurora.org>
Currently, the page table is flushed after the installation of each
individual page table entry. This is not terribly efficient since
virtual address ranges are often mapped with physically contiguous
chunks of page table memory. Optimize the map operation by factoring
out the page table flushing so that contiguous ranges of page table
memory can be flushed in one go.
Change-Id: Ie80eb57ef50d253db6489a6f75824d4c746314c7
Signed-off-by: Stepan Moskovchenko <stepanm@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
TTBR1 shouldn't currently be used at all. Any such usage is the result
of a bug. Configure TCR such that any usage of TTBR1 will generate a
fault, which will help catch such bugs earlier.
Change-Id: I74f2dc9580e5aed5d391debe23c7f5cc9fc1f672
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
MSM iommu devices' DT bindings defer from upstream
way. And hence, the debug driver doesn't support
to profile MSM iommu devices. Add that support.
Change-Id: I507d500785408b1d96f017648475933ee408b664
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
When a domain is attached for the first time, memory for its top level
of page tables is allocated. If it's a secure domain, this memory needs
to be assigned to a secure VMID. Currently, it's not being assigned to
a secure VMID, so when the domain is torn down and unassigned we get a
warning. Fix this by adding the missing assign call after initial
domain allocation.
Change-Id: I9ff2418749e768e17b4598f5f89e587ad7828d52
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, arm_smmu_unprepare_pgtable makes a hypervisor call to
unassign the memory being freed up. However, this is a sleeping call
and arm_smmu_unprepare_pgtable can be called from atomic context. Fix
this by making arm_smmu_unprepare_pgtable atomic in a similar manner to
arm_smmu_prepare_pgtable (put the addresses to be unprepared on a list
and unprepare them at the end of unmap).
Change-Id: Ifba40ca3ceb4aaf328a1a570e0ee6c39088075ba
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some IOMMU drivers (like arm-smmu) need to perform fixups to page table
memory when it's allocated and freed. Some callbacks were added for
this purpose ({prepare,unprepare}_pgtable) that are called from the
io-pgtable code after each allocation and before each free. However,
this approach is prone to bugs where new calls to allocate/free are
added without corresponding calls to the fixup callbacks. Furthermore,
allocation and free is often done from atomic context, so if the driver
needs to do non-atomic fixups during free they are out of luck since the
memory will be freed back to the system by the time control is turned
back over to the driver.
Adding yet another callback for non-atomic fixups would start to get
tedious, and would only increase the chance of missing callbacks as new
allocations/frees are added. Instead, fix this by allowing all page
table memory allocation and free to be delegated entirely to the driver.
Fall back to {alloc,free}_pages_exact by default if the driver doesn't
need any special handling of page table memory.
Change-Id: I0361bb81e25ff5ad4ef93a45330a35af47bc6013
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, arm_lpae_split_blk_unmap is assuming that the size to be
remapped is the block size of the next level. However, optimizations
have been made to the unmap code that result in the entire remaining
size being passed in, rather than just doing it one block at a time,
which breaks arm_lpae_split_blk_unmap.
Fix this by overriding the size passed in to be the block size of the
next level.
Change-Id: Ifce5b2e07dc15aba3cd37b7ac249e00decd2923f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful to perform tests on secure domains. Add support for
this with a "secure_attach" file that will also set the
DOMAIN_ATTR_SECURE_VMID to the value written to the file.
Example usage:
# cd /sys/kernel/debug/iommu/tests/<device>
# echo 0xA > secure_attach
# echo 0x1000,0x1000,0x1000,1 > map
# echo 0x1000,0x1000 > unmap
# echo 0 > secure_attach
Change-Id: I0181e360f3e98e0e47a3af8c6adb71c9c4a3c9b5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful to know a domain's secure VMID value. Add this
information to the attach info debugfs file.
Change-Id: Ic8a73000b8337639f95376b13ef58a0e2a089dde
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
