When using the client fault handler, set resume to terminate the
faulting transaction similar to the way of the default fault
handler.
Its possible that if the client fault handler doesn't "fully"
handle the fault, there would be a stream of page faults on the
same address since we would keep on retrying the same
transaction.
Change-Id: Id7aa439fd90aa5b63364c8ac270593330c55d672
Signed-off-by: Shrenuj Bansal <shrenujb@codeaurora.org>
Allow the creation of privileged mode mappings, for stage 1 only.
Change-Id: Ie91168012b61c0774d840a35b3ed2f18b834873f
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
It's not uncommon for us to take a flood of context faults when
something terrible happens. This usually brings down the system due to
excessive logging, which makes it difficult to debug the actual root
cause of the issue. Fix this by rate-limiting the output from the
context fault handler.
Change-Id: I1fe83bbbc7cc7aa2cf6303c6c7d2bf5733f63557
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
DMA allocations with no kernel mapping are likely
protected as well. Stale highmem mappings in such cases
with cache-able attributes may lead to speculative fetch
,as highmem mappings are not cleared out.
Hence flush out unused highmem mappings explicitly when
allocation request with no kernel mapping is requested.
Change-Id: Ic1de633c6364eaa1b6d5b0932f2cfe17d64d920e
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
The DMA_ATTR_NO_KERNEL_MAPPING is used to make sure that CMA
pages have no kernel mapping. Add support to make sure that
highmem pages have no mapping.
Change-Id: Ife76df126ecfedf0dba81a35e0de8a1787355b3d
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
Currently, there are use cases where not having any kernel
mapping is required; if the CMA memory needs to be used as
a pool which can have both cached and uncached mappings we
need to remove the mapping to avoid the multiple mapping
problem. Extend the dma APIs to use the DMA_ATTR_NO_KERNEL_MAPPING
with CMA. This doesn't end up saving any virtual address space
but the mapping will still not be present.
Change-Id: I64d21250abbe615c43e2b5b1272ee2b6d106705a
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
With the dma attribute 'SKIP_ZEROING', dma allocation skips memset and
also cache flush operations. This leaves dirty lines in the caches which
when evicted later would cause issue to the allocated buffer. For some
clients this may be okay but other clients need to do the actual sync.
Switch the code to only skip __dma_clear_buffer if both
DMA_ATTR_SKIP_ZEROING and DMA_ATTR_SKIP_CPU_SYNC are specified. If only
DMA_ATTR_SKIP_ZEROING is specified, just skip the zeroing but still do the
cache operations.
Change-Id: I81dd6bbeed7ec7f1dce28cd8d0aba7c8900ec550
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
The DMA framework currently zeros all buffers because it (righfully so)
assumes that drivers will soon need to pass the memory to a device.
Some devices/use case may not require zeroed memory and there can
be an increase in performance if we skip the zeroing. Add a DMA_ATTR
to allow skipping of DMA zeroing.
Change-Id: Ib78a19cb341c3c441f91d5b004c6375c80c10413
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Some SMMU operations (like register access and ATOS) require downstream
buses to be enabled. We've been getting lucky for a long time since
other clients have been voting for the buses in question, but recently
we've seen some unclocked accesses due to our missing vote. Add support
for bus scale voting to the driver. The voting parameters will be
provided later in the per-target device tree SMMU configuration.
Change-Id: I6b31c887ad817a6628f4234622c73dee046d6442
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The subdirectories under "attach" currently get a UUID appended to their
name to ensure that there won't be any collisions when multiple domains
are attached to the same device. This is only needed for dynamic
domains since those are the only types of domains that can be attached
to the same device concurrently. Only do the uniquification for dynamic
domains since the resulting directory name is a bit messy with the UUID.
Change-Id: Ia86b192e5b9d6a63132228fa86549a2977ec74a4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There's some non-deterministic latency incurred by enabling clocks and
regulators. Clocks and regulators are left on for atomic domains, which
makes them a good choice for profiling the page table management code.
Use an atomic domain for map/unmap profiling to make things more
deterministic.
Change-Id: Ib9fa48699c9d6009dd3f3b322002e2fb0f2ff99c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The iommu domain must be detached before the pagetable is freed,
if any TLB maintenance is needed it should be done by
iommu_detach_device().
Change-Id: I119d0e127e122de1263c80c73d40ff6edf16c875
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
The pagetable memory for a domain is freed by free_io_pgtable_ops(), which
is only called from arm_smmu_destroy_domain_context(). This function also
cleans up the hardware's context bank state and it is called during
iommu_device_detach(). Because dynamic domains don't have context bank
state, they don't call arm_smmu_destroy_domain_context() and thus their
pagetable memory was leaked.
Fix this by instead calling free_io_pgtable_ops() during
iommu_domain_destroy(). This ensures that the pagetable is freed for all
domains.
Change-Id: Id349fdc7df673cd4513468f23dcc95a9281d2657
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
We cannot call unmap directly if the map_sg fails partially as the tlb
invalidate functions need to enable/prepare clocks, which require
non-atomic context. Let map_sg return the failure and handle this when
we are out of the atomic context.
Change-Id: I6401c1e281850aeda27e32524cae34324045f762
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
It can be useful to read and modify SMMU settings interactively. Add a
debugfs interface for the recently added iommu_reg_read and
iommu_reg_write functions for this purpose.
Change-Id: If3c9bf1833dc950c7ca1e2ce1a373bcaa31ebcf0
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Implement the register reader and writer functions so that clients can
peek and poke their SMMU's registers.
Change-Id: I9dac59a16b42fec6b2a7639603289911f88b067d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It might be useful for IOMMU clients to peek and poke at their IOMMU's
registers, but knowing how to access those registers is really the job
of the IOMMU driver (it might need to enable specific clocks and
regulators, for example). Provide an API to read and write IOMMU
registers that can be implemented by the driver.
Change-Id: I5b2f19225f8bd258278780ff24b4ea96460857aa
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
For all but the last level of the page tables, we
need to call unmap with supported IOMMU pages sizes.
However currently to get the maximum page size that can be
unmapped, we are passing the original size parameter,
instead of the remaining size to be unmapped, and that can
end up unmapping more than what is intended.
Fix this by passing the remaining size parameter to
find the correct IOMMU page size.
Change-Id: Id4bd188ff6a3b4b7d34ba43ae6a61efb3c65b281
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Call the correct function to free the io_pgtable_ops when
there is an error.
Change-Id: Ib01041f874a203b18265b46b56b375c76c0ab211
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
iommu_trigger_fault now passes a flag value when triggering faults. Use
that value instead of hard-coding a translation fault.
Change-Id: Iac9b97cebef99e260641bde70c1ddfaedd0be502
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
As per the hardware spec, the SMMU must be halted during programming of
the implementation defined registers. Add the missing halt.
Change-Id: Id28b546e5b5b646c263d7431a32f3c95ed3f2042
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
iommu_trigger_fault now accepts a flags argument which can be handled by
the individual IOMMU drivers. Take the value written to the debugfs
file by the user and pass it as the flags value.
Change-Id: I803e6585aea35fbb0f6e03e4279587b36a7bad92
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
IOMMU drivers might want more control over the types of faults being
triggered with iommu_trigger_fault. Add a flags parameter that can be
used to provide more control over the types of faults that can be
triggered.
Change-Id: I2f21b383437430e957ab52070d3575e8cb3dee90
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
A client fault handler flag was recently introduced to indicate external
faults. Use it.
Change-Id: Ib9ea59befc3c8324de4fc94a0dbe4acafaa6f9b8
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some IOMMUs (like the ARM SMMU) can raise "external faults" in addition
to translation and permission faults. Add an IOMMU fault flag so that
this fault type can be propagated to client fault handlers.
Change-Id: Ia476ed6c727f5f01011747f30b3a74ebf0fc07f7
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Simultaneous assign or unassign on the same domain during map and unmap
operations manipulate the same pte_info lists and thus can ask hypervisor
to assign the same pte address twice which results in double mapping error.
The lists(pte_info_list and unassign_list) needs to be locked before
traversing it for making the assign/unassign calls.
Change-Id: Iebe3d703d80918af4ab18208f4b5a010c4c14791
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
When unmapping 2MB mappings, which are 2MB aligned, the smmu driver
is leaking the 3rd level page tables.
Fix this leak by updating __arm_lpae_free_pgtable so that it no
longer leaks leaf table entries.
To reproduce this leak simply map and unmap a non-block 2MB mapping
which is 2MB aligned.
Change-Id: Ibdbdb084ceb8d03ebe0a04e8777e3eb9419e9b87
Signed-off-by: Liam Mark <lmark@codeaurora.org>
Currently, the page table is flushed after the installation of each
individual page table entry. This is not terribly efficient since
virtual address ranges are often mapped with physically contiguous
chunks of page table memory. Optimize the map operation by factoring
out the page table flushing so that contiguous ranges of page table
memory can be flushed in one go.
Change-Id: Ie80eb57ef50d253db6489a6f75824d4c746314c7
Signed-off-by: Stepan Moskovchenko <stepanm@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
TTBR1 shouldn't currently be used at all. Any such usage is the result
of a bug. Configure TCR such that any usage of TTBR1 will generate a
fault, which will help catch such bugs earlier.
Change-Id: I74f2dc9580e5aed5d391debe23c7f5cc9fc1f672
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
MSM iommu devices' DT bindings defer from upstream
way. And hence, the debug driver doesn't support
to profile MSM iommu devices. Add that support.
Change-Id: I507d500785408b1d96f017648475933ee408b664
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
When a domain is attached for the first time, memory for its top level
of page tables is allocated. If it's a secure domain, this memory needs
to be assigned to a secure VMID. Currently, it's not being assigned to
a secure VMID, so when the domain is torn down and unassigned we get a
warning. Fix this by adding the missing assign call after initial
domain allocation.
Change-Id: I9ff2418749e768e17b4598f5f89e587ad7828d52
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, arm_smmu_unprepare_pgtable makes a hypervisor call to
unassign the memory being freed up. However, this is a sleeping call
and arm_smmu_unprepare_pgtable can be called from atomic context. Fix
this by making arm_smmu_unprepare_pgtable atomic in a similar manner to
arm_smmu_prepare_pgtable (put the addresses to be unprepared on a list
and unprepare them at the end of unmap).
Change-Id: Ifba40ca3ceb4aaf328a1a570e0ee6c39088075ba
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some IOMMU drivers (like arm-smmu) need to perform fixups to page table
memory when it's allocated and freed. Some callbacks were added for
this purpose ({prepare,unprepare}_pgtable) that are called from the
io-pgtable code after each allocation and before each free. However,
this approach is prone to bugs where new calls to allocate/free are
added without corresponding calls to the fixup callbacks. Furthermore,
allocation and free is often done from atomic context, so if the driver
needs to do non-atomic fixups during free they are out of luck since the
memory will be freed back to the system by the time control is turned
back over to the driver.
Adding yet another callback for non-atomic fixups would start to get
tedious, and would only increase the chance of missing callbacks as new
allocations/frees are added. Instead, fix this by allowing all page
table memory allocation and free to be delegated entirely to the driver.
Fall back to {alloc,free}_pages_exact by default if the driver doesn't
need any special handling of page table memory.
Change-Id: I0361bb81e25ff5ad4ef93a45330a35af47bc6013
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, arm_lpae_split_blk_unmap is assuming that the size to be
remapped is the block size of the next level. However, optimizations
have been made to the unmap code that result in the entire remaining
size being passed in, rather than just doing it one block at a time,
which breaks arm_lpae_split_blk_unmap.
Fix this by overriding the size passed in to be the block size of the
next level.
Change-Id: Ifce5b2e07dc15aba3cd37b7ac249e00decd2923f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful to perform tests on secure domains. Add support for
this with a "secure_attach" file that will also set the
DOMAIN_ATTR_SECURE_VMID to the value written to the file.
Example usage:
# cd /sys/kernel/debug/iommu/tests/<device>
# echo 0xA > secure_attach
# echo 0x1000,0x1000,0x1000,1 > map
# echo 0x1000,0x1000 > unmap
# echo 0 > secure_attach
Change-Id: I0181e360f3e98e0e47a3af8c6adb71c9c4a3c9b5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful to know a domain's secure VMID value. Add this
information to the attach info debugfs file.
Change-Id: Ic8a73000b8337639f95376b13ef58a0e2a089dde
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful to query a previously set secure VMID. Add a domain
attribute getter for this.
Change-Id: Iaf14201d321161ce4ccc69b674a3b1247d6a7e94
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
When memory that is assigned a particular VMID is re-assigned
back to HLOS, it needs to get RWX permissions so that HLOS can
use it for data as well as instructions.
Change-Id: Ib5413861de877d9081bdf5bd397f528293deb6e8
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
The unmap debugfs file write handler is currently incorrectly parsing
the size to be unmapped (storing the IOVA as the size). Fix this by
parsing from the correct offset (just after the first comma).
Change-Id: I438851882052acc511c713706bc4f0cbf373353e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, when all of the 4K PTEs beneath a 2M table entry are
unmapped, that 2M table entry is left intact, even though it doesn't
point to any valid 4K mappings anymore. This results in a warning if a
subsequent block mapping lands on top of the dangling table entry, since
we require empty page table entries when we map. It also causes the
page at which that the stomped-on table was pointing to be leaked. Fix
this by keeping track of how many entries are currently mapped beneath a
table. When the map count goes to zero (in unmap), free up the page the
table is pointing at and zero out the table entry.
Change-Id: I470e6ffb2206a09fe7c24253e3fd64a744337a7f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful for debugging to know how much memory is being used for
IOMMU page tables. Add some dedicated allocation functions and a
debugfs file for this.
Change-Id: Id69b4b1b5df5dcc6c604eec3a12a894b8eab0eb6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we're creating an "iommu" debugfs directory from the
iommu-debug code. Other IOMMU modules might want to make use of this
same directory, so create it from the IOMMU framework code itself.
Change-Id: I679fdfc34ba5fcbd927dc5981438c6fabcfa3639
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Implement support for dynamic domain switching. This feature is
only enabled when the qcom,dynamic device tree attribute for an smmu
instance.
In order to use dynamic domains, a non-dynamic domain must first
be created and attached. This non-dynamic domain must remain
attached while the device is in use.
All domains must be attached before calling any mapping functions, such as
iommu_map(). This allows the pagetable setup to be set up during attach
based on the hardware configuration for the smmu.
Before attaching a dynamic domain, the DOMAIN_ATTR_CONTEXT_BANK must be
set to indicate which context bank registers should be used for
any register programming. Attaching dynamic domain doesn't cause
hardware register programming, but mapping operations may cause
TLBI operations. Additionally, the upstream driver or hardware may
do other register programming.
Because the arm-smmu driver assigns context banks dynamically, the
upstream driver should query DOMAIN_ATTR_CONTEXT_BANK on its non-dynamic
domain, to ensure the correct value is used for all dynamic domains
created.
To switch domains dynamically, the upstream driver or hardware
must program the TTBR0 and CONTEXTIDR registers with the values
from the DOMAIN_ATTR_TTBR0 and DOMAIN_ATTR_CONTEXTIDR attributes
for the desired domain. The upstream driver may also need to do
other hardware specific register programming to properly
synchronize the domain switch. It must ensure that all register
state, except for CONTEXTIDR and TTBR0 registers, is restored
at the end of the domain switch operation.
DOMAIN_ATTR_PROCID may be set to any value for each domain
before it is attached. This value is part of the CONTEXTIDR
register, but it is not used by the SMMU hardware. Setting a unique
value for this attribute in every domain can be useful for debugging.
Change-Id: Ib92d06db06832700bdf56265b169ccddfb192698
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
Some devices can support per-process iommu domains, which are used
asynchronously by the hardware which directly updates to the TTBR0
and CONTEXTIDR registers.
Add DOMAIN_ATTR_DYNAMIC to do indicate a domain may be used dynamically.
This attribute must be set before attaching, as it changes what
domain and hardware configuration is done by the iommu driver.
Before attaching any dynamic domains, the driver must first attach
a non-dynamic domain to do initial hardware configuration.
A simplified example:
void use_domain(struct iommu_domain *domain, struct job *job)
{
u64 ttbr0;
u32 contextidr;
iommu_domain_get_attr(domain, DOMAIN_ATTR_TTBR0, &ttbr0);
iommu_domain_get_attr(domain, DOMAIN_ATTR_CONTEXTIDR, &contextidr);
/*
* Schedule work on the hardware, which is time sliced between
* clients. Each client has a separate iommu domain and when
* a job is run, the hardware first programs TTBR0 and
* CONTEXTIDR to use the appropriate domain.
*/
submit_job(ttbr0, contextidr, job);
}
void init(void)
{
struct iommu_domain *master_domain, *dyn_domain1, *dyn_domain2;
int dynamic = 1;
master_domain = iommu_domain_alloc(bus);
dyn_domain1 = iommu_domain_alloc(bus);
dyn_domain2 = iommu_domain_alloc(bus);
iommu_attach_device(base_domain, dev);
iommu_domain_set_attr(dyn_domain1, DOMAIN_ATTR_DYNAMIC, &dynamic);
iommu_domain_set_attr(dyn_domain2, DOMAIN_ATTR_DYNAMIC, &dynamic);
iommu_attach_device(dyn_domain1, dev);
iommu_attach_device(dyn_domain2, dev);
while (keep_going) {
iommu_map(dyn_domain1, ...);
iommu_map(dyn_domain2, ...);
use_domain(dyn_domain1, job1);
use_domain(dyn_domain2, job2);
iommu_unmap(dyn_domain1, ...);
iommu_unmap(dyn_domain2, ...);
}
iommu_detach_device(dyn_domain2, dev);
iommu_detach_device(dyn_domain1, dev);
iommu_detach_device(master_domain, dev);
}
Change-Id: Ic4fa0a831751eb4b10fff5d9aec28a411856fbd1
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
Add fields to arm_smmu_cfg to store asid, vmid, and procid so that they
can queried easily.
Program the CONTEXTIDR register during attach so that the PROCID
attribute functions as expected.
Change-Id: I2e2b3926fbf021754e89edda9a6857d2e3a7138b
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
In the ARM SMMU architecture, pagetable programming is controlled
by the TTBR0 and CONTEXTIDR registers. The layout of these
registers varies depending on the pagetable format in use.
In particular, the ASID (address space ID) field is found in
CONTEXTIDR when using V7S format and in the top bits of TTBR0
for V7L and V8L.
Some drivers need to program hardware to switch domains on the
fly. These attributes allow the correct settings to be determined
by querying the domain rather than directly reading registers and
making assumptions about the pagetable format. The domain must be
attached before TTBR0 and CONTEXTIDR may be queried.
The PROCID attribute allows driver set a debug field in the
CONTEXTIDR register. This attribute may only be set before
attaching, but may be queried at any time. The SMMU hardware
doesn't use the contents of this field, but debug can be simpler
if each domain stores a unique value in it.
Change-Id: I175aa78fee02c3e4e0071496d9cc2b8841ff9e3c
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
The attachment tracking code adds a node to debugfs every time a client
attaches a domain to an IOMMU device. The problem is that clients can
start making those attachments during early boot, before iommu-debug
initializes (including setting up the root debugfs directory for the
attachment tracking), which means they get stuck in the root debugfs
directory. Trying to initialize iommu-debug earlier than all possible
IOMMU clients is tricky, so fix this by only installing debugfs entries
onces we've initialized, installing debugfs entries for any early
attachments at initialization time.
Change-Id: I8364015346105187e0c8f787fc2b4155d72b3584
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful during development to trigger faults. Add a debugfs
file to do so.
Change-Id: Ic7b304ef0d908ebd506979f0c87189e34d7dfc67
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
In order to facilitate debugging and development, implement the
.trigger_fault method from iommu_ops. This can be done on ARM SMMUv2 by
writing to the FSRRESTORE register. Do it.
Change-Id: Ia8339b54fbb9263d8cf23ff61c4615122316729a
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful to trigger an IOMMU fault during development and
debugging. Add support to the IOMMU framework to do so.
Change-Id: I908c9f5b52c6abe937f031de546d290027ba64b5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
