This patch adds a new function to the iommu_ops structure to allow an
OF device to be added to a specific IOMMU instance using the recently
merged generic devicetree binding for IOMMUs. The callback (of_xlate)
takes a struct device representing the master and an of_phandle_args
representing the IOMMU and the correspondong IDs for the new master.
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
IOMMU drivers must be initialised before any of their upstream devices,
otherwise the relevant iommu_ops won't be configured for the bus in
question. To solve this, a number of IOMMU drivers use initcalls to
initialise the driver before anything has a chance to be probed.
Whilst this solves the immediate problem, it leaves the job of probing
the IOMMU completely separate from the iommu_ops to configure the IOMMU,
which are called on a per-bus basis and require the driver to figure out
exactly which instance of the IOMMU is being requested. In particular,
the add_device callback simply passes a struct device to the driver,
which then has to parse firmware tables or probe buses to identify the
relevant IOMMU instance.
This patch takes the first step in addressing this problem by adding an
early initialisation pass for IOMMU drivers, giving them the ability to
store some per-instance data in their iommu_ops structure and store that
in their of_node. This can later be used when parsing OF masters to
identify the IOMMU instance in question.
Change-Id: Ie7051c90c0f2e790d016eb465f10ddda48b99cd1
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
[pdaly@codeaurora.org Resolve minor conflicts]
The ARM SMMU supports the IOMMU_NOEXEC protection flag. Add the
corresponding IOMMU capability.
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Some IOMMUs accept an IOMMU_NOEXEC protection flag in addition to
IOMMU_READ and IOMMU_WRITE. Expose this as an IOMMU capability.
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
When some part of bus_set_iommu fails it should undo any made changes
and not simply leave everything as is.
This includes unregistering the bus notifier in iommu_bus_init when
add_iommu_group fails and also setting the bus->iommu_ops back to NULL.
Signed-off-by: Heiko Stuebner <heiko@sntech.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
The IOMMU-API works on page boundarys, unlike the DMA-API
which can work with sub-page buffers. The sg->offset
field does not make sense on the IOMMU level, so force it to
be 0. Do some error-path consolidation while at it.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
A platform_driver does not need to set an owner, it will be populated by the
driver core.
Change-Id: I4382a4b82bba56cf5cd2c2bfe56cdbafeb647cb8
Signed-off-by: Wolfram Sang <wsa@the-dreams.de>
[pdaly@codeaurora.org Resolve minor conflicts]
Print time with ns precision.
Ex:
size iommu_map iommu_unmap
4K 43.021 us 99.479 us
Change-Id: Ib7f3db2be533c77adf0a20de557615a0e2433dd9
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Add a nr_iters file which specifies the number of runs to average
performance over. Alternatively, setting this to 1 allows the user
to collect raw data.
Change-Id: Ifab7215027b597fb2650ab2f0c5065bebe0b8c5d
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Allow compilation when CONFIG_ARM_DMA_USE_IOMMU is not
selected by providing necessary stub functions and fix few
mismatch in function declarations.
Following this, remove the changes introduced by
f3d8d1061fb0b146b3f5 ("msm: ipa: add empty implementation
for iommu functions") as they are no longer needed.
Change-Id: I04e3aa63407064e8d9c9550a5cb0a82ede899f00
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
The fault handler currently spits out a ton of information that can make
clients' eyes glaze over, and understandably so. One of the most
important pieces of information we obtain from the fault handler is
whether the faulting address is actually mapped (which we check with a
software table walk), but that information is currently not obvious (you
have to know that "soft iova-to-phys=0" means that the software table
walk failed). Fix this by adding a more obvious error message when we
detect an unmapped access.
Change-Id: Ib95a916baf3e73934642676de4852a550ae0f54b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We're currently waiting for up to 1 second for the TLB sync command to
finish. Unfortunately, this is long enough to trigger a watchdog bark
since the time between the watchdog pet timer interrupt and the bark
interrupt is only 1 second. Fix this by reducing the timeout to 500
milliseconds, which is still more than enough time for the TLB sync
command to complete.
CRs-Fixed: 929880
Change-Id: I467cd5fd9caa24b4f120a7fe8c8f57c73687006b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The SMMU halt needs to be accessed before regulator is disabled. So move
notification from REGULATOR_EVENT_DISABLE to REGULATOR_EVENT_PRE_DISABLE.
Regulator framework will notify client on other events too, return
NOTIFY_OK.
Change-Id: Ieed950418d5afc6f5b84b13131ef39964f3168e2
Signed-off-by: Jaydeep Sen <jsen@codeaurora.org>
As per the spec, FRSYNRA contains the SID in the lower 16 bits.
Currently we're masking off all but the lower 8 bits, so we could be
losing bits when we print SIDs. Fix this by using a 16 bit mask.
Change-Id: I4a16c169a52d31b43b2f850a5558eecb64b93902
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Before SMMU is powered down, SMMU needs to be in
idle state prior to power collapse. When 'halt' is
received by SMMU, it ensures that no new requests
enters and all outstanding requests are completed
and generates an acknowledgment for halt request.
Add an option to register a notifier on regulator
so that SMMU can be halted/resumed when regulator
is powered down/up. Some of the targets have built
in provision in hardware to halt SMMU when there
are no on going transactions. We should keep this
option disable for those targets.
Change-Id: Ia2f8a934a9d64daefdacd517eb22f09de5eeb273
Signed-off-by: Susheel Khiani <skhiani@codeaurora.org>
We're currently disabling clocks/power immediately after writing to
FSRRESTORE, but if the fault is not handled before we disable the
clocks/power then the fault could be lost. Add a 1-second sleep after
writing to FSRRESTORE to give the interrupt time to fire before turning
clocks/power back off.
Change-Id: Ieeba49686da5da92e43f03c3c593166592184433
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, in the arm_lpae_map_sg early return path (when neither READ
nor WRITE permissions were specified), we don't store the "partially
mapped" size in the size output parameter, as is expected. This could
result in someone up the stack using an uninitialized variable. Fix
this by storing 0 in *size when there's nothing to do.
Change-Id: I3b0952aa5bcc2810843b7c513c32c3d665169e40
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The return value of the .map_sg callback is currently undocumented.
Document it.
Change-Id: I195cbfb78eaf173d916dd03190f89b2fe04e144e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Unhandled context faults should be fatal unless explicitly requested
otherwise by the client. Implement the DOMAIN_ATTR_NON_FATAL_FAULTS
attribute for this purpose.
Change-Id: I414fcc6000a6c47b3cbf77b1098c8b7895cbb20d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
During development, any IOMMU faults most likely need to be debugged
immediately. Allowing the system to continue to run has a tendency to
mask the original problem, which makes debugging more difficult. Some
drivers might want to make all faults fatal by default, unless
explicitly requested otherwise by a client. Add a domain attribute for
this.
Change-Id: I454b143ea3cae1f3de36e0888afd15936aa63ce1
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The clock and regulator APIs can fail, but we're ignoring their return
values. This results in strange code paths where we misuse the
API (e.g. we enable clocks after a failed prepare, which makes the clock
driver yell). Fix this by checking all return values and propagating
the errors up to clients as needed.
CRs-Fixed: 924600
Change-Id: Ib6cb3afac1f5472f3a5abb5c251765344fe1a7a9
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The device config probe messages are for informational purposes only.
Move them to debug level to avoid spamming the boot log unnecessarily.
CRs-Fixed: 918584
Change-Id: If9085e2165f26b66ff94b258894f01471be82a5b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we only take one sample per buffer size per API when doing
profiling. This results in high run-to-run variance in the results.
Use more iterations to help smooth this out.
Change-Id: I3779007a2f69ef79b573285b2422554f42dda99f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Secure domains do different amounts of work than non-secure domains, so
we need to be able to profile them separately. Add another debugfs
file, "secure_profiling", that does the same thing as the "profiling"
file.
Change-Id: I6968036bfdcc63449f27336771b3956a2b7754f6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
secure_buffer.h makes use of the scatterlist structure, but doesn't
include the header file that defines it. Until now we've been getting
lucky that all users of secure_buffer.h have been including the
scatterlist header file themselves, but this might not always be the
case. Add the missing include.
Change-Id: If7598c26a1ef3551132c6eb62ac9bd6d447654d0
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The hypervisor assign call has substantial overhead, so calling it every
time we need to allocate some page table memory is currently killing
performance. Fix this by hanging on to secure page table memory when
it's freed, keeping it in a pool of pre-assigned memory for future
allocations.
Change-Id: Iefa45e6bc23e841d168be4f17c1f669f2e932a3b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We've been slowly littering arm-smmu.c with forward declarations all
over the place, moving them around when they're not high enough in the
file, etc. Just move them all to the top of the file once and for all.
Change-Id: I8f2f075546f287db17f1b6e3b08390296cb9b9ff
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Simultaneous maps and/or unmaps will lead to a race condition,
where if the assign list has multiple entries and one call is
in the middle of an assign operation for one entry, and the other
call removes the next element from the list and queues itself for
the assign operation. When the first call returns from assign, and
the list is empty, it will return back to the caller without
completing the assign on all the ptes allocated for that mapping.
This results in a page fault when the client is actually accessing
the buffer.
Protect the assign and unassign lists with a mutex to ensure that
the operation is serialized and the control does not return back
to the client before all the ptes have been correctly assigned.
Change-Id: Idaf894671fee9fc9c8d339bfa17344b68dd0ad77
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Currently there's a check to make sure that a mapping isn't being
installed on top of an existing leaf mapping. However, mapping on top
of an existing table mapping is silently ignored. Make the check more
strict by testing for any valid page table entry (table, block, or page
descriptor) and by making any overlap fatal.
Change-Id: Ibdffbfdb2b44cf9da5044c2c1828a32979a39760
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
When we brought back hardware ATOS support [94bc89042be5cb:
"iommu/arm-smmu: Resurrect hardware iova-to-phys"] we added some macros
back that were never deleted in the first place. Clean up the redundant
macros.
Change-Id: I0129313fa4132f45fe5805e9ed3477ef7819e70e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The MICRO_MMU_CTRL register is used for halt control, but it also
contains some important configuration bits. Currently, when we request
a halt, we're wiping out all of the other bits in the register, losing
any previous configuration. Fix this by reading the value and OR'ing in
the bits we need.
Change-Id: Iafdc5b2a344916dce9bac7f958a153dbee042ca8
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some SMMUs may not have a dedicated gdsc for it.
Allow for the enable code to prepare clocks and vote
for bus requests even if there is no gdsc provided.
Change-Id: Iaedfb73d8ae7f12895472893468e5c32d2bf2462
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
When using the client fault handler, set resume to terminate the
faulting transaction similar to the way of the default fault
handler.
Its possible that if the client fault handler doesn't "fully"
handle the fault, there would be a stream of page faults on the
same address since we would keep on retrying the same
transaction.
Change-Id: Id7aa439fd90aa5b63364c8ac270593330c55d672
Signed-off-by: Shrenuj Bansal <shrenujb@codeaurora.org>
Allow the creation of privileged mode mappings, for stage 1 only.
Change-Id: Ie91168012b61c0774d840a35b3ed2f18b834873f
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
It's not uncommon for us to take a flood of context faults when
something terrible happens. This usually brings down the system due to
excessive logging, which makes it difficult to debug the actual root
cause of the issue. Fix this by rate-limiting the output from the
context fault handler.
Change-Id: I1fe83bbbc7cc7aa2cf6303c6c7d2bf5733f63557
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
DMA allocations with no kernel mapping are likely
protected as well. Stale highmem mappings in such cases
with cache-able attributes may lead to speculative fetch
,as highmem mappings are not cleared out.
Hence flush out unused highmem mappings explicitly when
allocation request with no kernel mapping is requested.
Change-Id: Ic1de633c6364eaa1b6d5b0932f2cfe17d64d920e
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
The DMA_ATTR_NO_KERNEL_MAPPING is used to make sure that CMA
pages have no kernel mapping. Add support to make sure that
highmem pages have no mapping.
Change-Id: Ife76df126ecfedf0dba81a35e0de8a1787355b3d
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
Currently, there are use cases where not having any kernel
mapping is required; if the CMA memory needs to be used as
a pool which can have both cached and uncached mappings we
need to remove the mapping to avoid the multiple mapping
problem. Extend the dma APIs to use the DMA_ATTR_NO_KERNEL_MAPPING
with CMA. This doesn't end up saving any virtual address space
but the mapping will still not be present.
Change-Id: I64d21250abbe615c43e2b5b1272ee2b6d106705a
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
With the dma attribute 'SKIP_ZEROING', dma allocation skips memset and
also cache flush operations. This leaves dirty lines in the caches which
when evicted later would cause issue to the allocated buffer. For some
clients this may be okay but other clients need to do the actual sync.
Switch the code to only skip __dma_clear_buffer if both
DMA_ATTR_SKIP_ZEROING and DMA_ATTR_SKIP_CPU_SYNC are specified. If only
DMA_ATTR_SKIP_ZEROING is specified, just skip the zeroing but still do the
cache operations.
Change-Id: I81dd6bbeed7ec7f1dce28cd8d0aba7c8900ec550
Signed-off-by: Taniya Das <tdas@codeaurora.org>
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
The DMA framework currently zeros all buffers because it (righfully so)
assumes that drivers will soon need to pass the memory to a device.
Some devices/use case may not require zeroed memory and there can
be an increase in performance if we skip the zeroing. Add a DMA_ATTR
to allow skipping of DMA zeroing.
Change-Id: Ib78a19cb341c3c441f91d5b004c6375c80c10413
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Some SMMU operations (like register access and ATOS) require downstream
buses to be enabled. We've been getting lucky for a long time since
other clients have been voting for the buses in question, but recently
we've seen some unclocked accesses due to our missing vote. Add support
for bus scale voting to the driver. The voting parameters will be
provided later in the per-target device tree SMMU configuration.
Change-Id: I6b31c887ad817a6628f4234622c73dee046d6442
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The subdirectories under "attach" currently get a UUID appended to their
name to ensure that there won't be any collisions when multiple domains
are attached to the same device. This is only needed for dynamic
domains since those are the only types of domains that can be attached
to the same device concurrently. Only do the uniquification for dynamic
domains since the resulting directory name is a bit messy with the UUID.
Change-Id: Ia86b192e5b9d6a63132228fa86549a2977ec74a4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There's some non-deterministic latency incurred by enabling clocks and
regulators. Clocks and regulators are left on for atomic domains, which
makes them a good choice for profiling the page table management code.
Use an atomic domain for map/unmap profiling to make things more
deterministic.
Change-Id: Ib9fa48699c9d6009dd3f3b322002e2fb0f2ff99c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The iommu domain must be detached before the pagetable is freed,
if any TLB maintenance is needed it should be done by
iommu_detach_device().
Change-Id: I119d0e127e122de1263c80c73d40ff6edf16c875
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
The pagetable memory for a domain is freed by free_io_pgtable_ops(), which
is only called from arm_smmu_destroy_domain_context(). This function also
cleans up the hardware's context bank state and it is called during
iommu_device_detach(). Because dynamic domains don't have context bank
state, they don't call arm_smmu_destroy_domain_context() and thus their
pagetable memory was leaked.
Fix this by instead calling free_io_pgtable_ops() during
iommu_domain_destroy(). This ensures that the pagetable is freed for all
domains.
Change-Id: Id349fdc7df673cd4513468f23dcc95a9281d2657
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
We cannot call unmap directly if the map_sg fails partially as the tlb
invalidate functions need to enable/prepare clocks, which require
non-atomic context. Let map_sg return the failure and handle this when
we are out of the atomic context.
Change-Id: I6401c1e281850aeda27e32524cae34324045f762
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
It can be useful to read and modify SMMU settings interactively. Add a
debugfs interface for the recently added iommu_reg_read and
iommu_reg_write functions for this purpose.
Change-Id: If3c9bf1833dc950c7ca1e2ce1a373bcaa31ebcf0
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Implement the register reader and writer functions so that clients can
peek and poke their SMMU's registers.
Change-Id: I9dac59a16b42fec6b2a7639603289911f88b067d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It might be useful for IOMMU clients to peek and poke at their IOMMU's
registers, but knowing how to access those registers is really the job
of the IOMMU driver (it might need to enable specific clocks and
regulators, for example). Provide an API to read and write IOMMU
registers that can be implemented by the driver.
Change-Id: I5b2f19225f8bd258278780ff24b4ea96460857aa
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
For all but the last level of the page tables, we
need to call unmap with supported IOMMU pages sizes.
However currently to get the maximum page size that can be
unmapped, we are passing the original size parameter,
instead of the remaining size to be unmapped, and that can
end up unmapping more than what is intended.
Fix this by passing the remaining size parameter to
find the correct IOMMU page size.
Change-Id: Id4bd188ff6a3b4b7d34ba43ae6a61efb3c65b281
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
