Thulium v1 has an ATOS hardware errata that requires that we call into
TZ to do some fixups. Add a DT option to enable this workaround.
Change-Id: Ida2fecf1b40ba0f37c9cacc4296b0e8e46db071c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some targets require calling into TZ for various bits of SMMU
configuration. Add a library to provide a clean interface for such
configuration.
Change-Id: I1dc5cd21d4a09e321039d69cc760eaf6f356c6cf
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Because of a hardware errata, arm_smmu_iova_to_phys_hard will soon be
making a TZ call to do a workaround. However, we're currently using a
spinlock to ensure atomicity of ATOS due to another hardware errata, but
scm_call is a sleeping function, so this results in a sleeping BUG. Fix
this by making the atos lock a mutex instead of a spinlock.
This isn't exactly correct since iommu_iova_to_phys itself might be
called from atomic context, so we really shouldn't be taking a mutex.
However, we don't seem to have any use cases where it will be called
from atomic context, but we should revert all of this and go back to a
spinlock as soon as this hardware errata goes away (which will happen
when Thulium v1 dies).
Change-Id: I61ea37bb3e6989fe5db43c4e828fc6473885db1e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We're currently mapping a page in arm_smmu_flush_pgtable without ever
unmapping it. Fix this by calling dma_unmap_page on the returned dma
address. Since the only reason we're calling dma_map_page is to make
sure it actually gets flushed out to RAM, we can just call
dma_unmap_page immediately following the map.
Without this, eventually swiotlb runs out of memory and starts printing
things like:
arm-smmu d00000.arm,smmu: swiotlb buffer is full (sz: 128 bytes)
Change-Id: I69421de6e2189a16cd88f225e2698cd88d669dff
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
To aide in debugging, print the value of the IDR0 register when no
translation support is found, which would generally indicate a hardware
bug or missing clocks/power.
Change-Id: I22dd44fc30012c72a7d04088ab079076b6837f7e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Since we use dma_map_page() as an architecture-independent means of
making page table updates visible to non-coherent SMMUs, we need to
have a suitable DMA mask set to discourage the DMA mapping layer from
creating bounce buffers and flushing those instead, if said page tables
happen to lie outside the default 32-bit mask.
Change-Id: Ic5accfc4da5dff76123d7c524a9da77c9d88a847
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Patch-mainline: iommu @ 5 Mar 2015 18:56:36 +0000
[mitchelh@codeaurora.org: context differences]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Thulium v1 has a hardware bug that requires us to zero out page table
entries completely rather than just setting the valid bits to 0. The
previous workaround for this [165066b85d: "iommu/arm-smmu: work around
transaction hardware bug"] was actually not quite right since it still
allowed the page frame number to be set in the page table entry. Fix
this by actually zero'ing out the pte.
Change-Id: I37537874ea4c5d1c00db7de65f70edd93ce63b90
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Thulium has a hardware errata that requires that we avoid using ASID 0.
Implement the workaround in the SMMU driver.
Change-Id: Ia44ab1b385450994641c5f5ccb3dba4a462a8033
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.
Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.
Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Change-Id: I1f3dd2c3cf67b3db40ee1793580d6af5fec1247d
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Git-commit: 315786ebbf
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[mitchelh: fix existing callers and implementations of
iommu_{map,unmap}_range to match the new function names and APIs,
maintaining stubs for the old API so that out-of-tree modules can
continue to compile]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We're currently wiping out the `far' variable in our context fault
handler when CONFIG_64BIT is enabled. Fix the bug.
Change-Id: I86e57c7de8bfc10118f902d6702d3b9059a8762a
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, we fall back to a software table walk if ATOS fails. This
can mask problems that might be harder to debug later. Rather than
doing a software table walk and returning success, just print the result
of the software table walk to the log and return failure.
Change-Id: I28335b33977e60aea43d2c0ee1f4571a14a5e191
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we don't print a whole lot in the context fault handler. Make
it more verbose to facilitate debugging.
Change-Id: I4c0570184e0129c1d0c40f5d632a412de7c81cd4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We need to make sure the TLBIALL is actually done before proceeding with
the ATOS since the ATOS won't succeed if the TLB hasn't been
invalidated. Add a TLBSYNC to ensure that it finishes.
Change-Id: I55836039d7ded5ba8889bc81ee474624e98ef491
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We need to do some sleeping operations (like enabling clocks) in the
global fault handler. Move to a threaded handler to avoid BUGs due to
sleeping in an atomic context.
Change-Id: I88eed743bfad4cc33daf3b9f81255587c72b0167
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some hardware requires special fixups to recover from address size
faults. Rather than applying the fixups add an option to just BUG since
address size faults are due to a fundamental programming error from
which we don't care about recovering anyways.
Change-Id: Ibb70e4ec00683562dae9f3239b286daa5deabd45
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There is a hardware errata that could result in bus hang during context
fault processing. The work around given by the hardware team is to
issue a TLBSYNC and terminate the transaction. Implement the
workaround and provide a DT option to activate it.
Change-Id: Iede30ff68676188af1249fd2fd776bc84f224dc6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, we program the TCR2.SEP to be equal to the input address
size. This makes it impossible to use the top bit of the input address
space since that causes TTBR1 to be selected, which we don't program or
use. Since we don't even use sign-extended addresses, fix this by
always using the "no-sign" value for TCR2.SEP as specified by the ARM
SMMUv2 spec.
Change-Id: I9155b62fe59c66185c6b9fba7f8bba8798b45785
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we're setting page table cacheability to "inner shareable"
when coherent table walking is enabled, but it should actually be "outer
shareable". Fix this.
Change-Id: I7939bb75da8327f01a2fcd941a0d8358de6f0bc5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently when an SMMU client accesses an unmapped address the SMMU does
a bypass and propagates the request. This can make it difficult to
debug problems when clients are accessing bogus addresses. Enable the
sCR0 bit that causes the SMMU to raise an unidentified stream fault
rather than falling back to bypass when no mapping is found.
Change-Id: Ia0e40010c79b8a75954ae13ae60aca2d82ff76b6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
On some targets, context banks can be dedicated for use by other
execution environments outside of Linux. However, we currently assume
that we own all of the context banks during SMMU initialization in
arm_smmu_device_reset. This can mess up the other execution
environments since we are trampling on their SCTLR, SMR, and S2CR.
Provide a DT option to skip this initialization altogether, since the
other execution environment should have already initialized the SMMU
anyways.
Change-Id: I0ed4cbbcdad596f9201f83cb7d0e28a289e18a6e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Context interrupts can call domain-specific handlers which might sleep.
Currently we register our handler with request_irq, so our handler is
called in atomic context, so domain handlers that sleep result in an
invalid context BUG. Fix this by using request_threaded_irq.
This also prepares the way for doing things like enabling clocks within
our interrupt handler.
Change-Id: I2f12f2aa234b62532807a208cd70f2d705083343
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some hardware is capable of retaining register values during power
collapse. Add an option (configurable via DT) to enable this feature.
This is implemented by always enabling/disabling regulators every time
clocks are enabled/disabled.
Change-Id: I89d9a4f4a2eb29f0868b309d55a77cc4ed50e22e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Hardware requires that the ATOS command be issued with a single 64-bit
write instead of two 32-bit writes as we're currently doing. Fix this.
Change-Id: I43104c89c2f27b75d1176c8cbcd214666321244f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Thulium has a hardware errata that requires that the SMMU be halted and
that a TLBIALL be issued before the ATOS command. Add a DT option that
implements this workaround.
Change-Id: Ic40c7b93d64eebb97fe77082d8335debab624af1
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some hardware prohibits multiple ATOS commands from being issued
concurrently. Add a lock to avoid concurrent ATOS commands from being
issued.
Change-Id: Ie728464f8f03b3d60807a75b9e7c2b50ab62fa43
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The meaning of the arguments to readl_poll_timeout_atomic have changed
since the arm-smmu code was written that uses it. Sync up to the new
API semantics.
Change-Id: Ibdcd53fb61303e1261f7f6378b3395700ce38b6e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we do a STLBIALL when we initialize the SMMU. However, in
some configurations that register is not supposed to be touched and is
marked as "Secure only" in the spec. Rip it out.
Change-Id: I3a6e3f171f8f9fd82257270e3a7202ecbc8bd6ad
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
When running under a hypervisor, the number of context banks visible to
Linux could be different from what we see when running standalone. This
is problematic since the context bank interrupts are hard-coded into the
DT configuration, and we'd like to share that configuration whether
we're running under a hypervisor or not. Remove a warning when a
mismatch in the number of CBs and CB interrupts is detected and don't
return failure while probing.
Change-Id: I9271e4e04ecafe61f4bb12d1e1148cb2bcec2439
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some clients need to get the base address of the page table for a given
IOMMU domain. This functionality is provided by the
DOMAIN_ATTR_PT_BASE_ADDR domain attribute. Implement it in the ARM SMMU
driver.
Change-Id: Ibe4b11a6eaf2ca42aec326c764777373cff769cc
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we're repurposing the definitions used for the TTBCR2.PASize
field for TTBCR2.SEP since they conveniently have the same values.
However, this might look like a bug at first glance to the casual
passer-by, appearing that we're using the wrong bit definitions for the
field. For example, we're using TTBCR2_ADDR_32 to indicate that the
Sign Extension Bit should live at 31 (not 32 as the name of the macro
might imply). Reduce cumulative human cognitive load by adding some
definitions specifically for the SEP field.
Change-Id: Ia406951499453e2badca42a1cc4cdbc566af4dab
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently there's no way to clear DOMAIN_ATTR_COHERENT_HTW_DISABLE=0 if
you've ever set it. Fix this.
Change-Id: I10d3bb2f72a94b2b91d4a0500e0e2d9462efcf36
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
According to the IOMMU API, a domain doesn't necessarily need to be
attached to an IOMMU device before setting up mappings on that domain.
However, in the ARM SMMU driver we currently make the assumption that
the domain has already been attached to an SMMU. We use this assumption
for the following information:
(1) To know whether this is a stage1 or stage2 mapping
(2) To know whether we need to do an invalidate
(ARM_SMMU_OPT_INVALIDATE_ON_MAP)
(3) Address sanity checks based on the stage-1 and stage-2
sizes (which are determined by the SMMU configuration)
(4) To call the correct cache maintenance routines based on the SMMU
device pointer
Handle (1) by assuming that the mapping is stage1 by default. If
there's an SMMU attached we can check if this is a stage1 or stage2
mapping. This works for us at the moment because no one is setting up
stage2 mappings in the kernel.
Handle (2) by not invalidating on map if the domain isn't attached to an
SMMU since we shouldn't have anything to invalidate in that case
anyways.
Handle (3) by skipping the sanity checks for unattached domains.
Handle (4) by using the low-level caching APIs.
Currently we don't even free up the context bank when the device is
detached, presumably so that we don't have to set it up again on
re-attach. This limits the number of domains that can be attached to an
SMMU, even if they detach from it before the next one attaches. Remove
this 1-to-1 domain-to-SMMU assumption by freeing up the "domain
context" (context bank allocation, etc) on detach.
CRs-Fixed: 752812
Change-Id: I990a11ed52381ce4ecea0f82b4c6e2d54e08f2d6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we expose a domain attribute to disable/enable coherent
hardware table walks. However, while we're building page tables, we
flush the cache based on the SMMU's capability to perform coherent
hardware table walks, rather than flushing based on whether coherent
hardware table walks are actually enabled for the domain. We also allow
domains that have asked to enable coherent hardware table walks to be
attached to SMMU devices that don't support it.
Fix these issues by doing validation of the domain attributes against
the SMMU device to which the domain is attached. We need to do this at
domain attribute set time and at attach time since domain attributes can
be set before and after they are attached to a device.
This also allows us to drop the check for whether the SMMU supports
coherent table walks in the page table flushing code. We can simply
check the domain attributes instead of checking both the domain
attributes and the SMMU features.
CRs-Fixed: 752812
Change-Id: Ibee9316a340f481664f5fb7757a24f136e0ed74c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We're currently taking the upper 52 bits from the CB_PAR register and
using them for the output physical address. That doesn't make sense for
a few reasons, not the least of which is the fact that physical
addresses are only 48 bits on ARM64 and even less on ARM32. Also, when
using V7L and V8 descriptor formats the top byte of the CB_PAR register
contains the memory attributes for the translation. Fix this by masking
out everything above the valid physical address bits with PHYS_MASK.
Change-Id: I96096a6515dc42025d3134933a90b072e5153968
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We recently increased the maximum number of stream IDs from 8 to 16 in
[85fac623740fc: "iommu/arm-smmu: increase maximum stream ids"].
However, we can actually have many more than 16. Go ahead and increase
this limit to accomodate the maximum number of stream IDs as outlined in
the SMMUv2 spec.
Change-Id: I23cee7ad78f0d388e339a4c8a1058f0674f3edc3
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We support way more than 8 SIDs but currently error out if we get more
than that. Fix this.
Change-Id: I50e70b07ffbee783d3e6582fa7d9a175a1b26fe5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We currently don't check whether register_smmu_master while parsing the
device tree and adding device masters. Add error checking to facilitate
debugging and avoid probing of bogus devices.
Change-Id: I79c70d193c4705c21ea6a2e03ee048e7a4813b71
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we only program the VMID (in the CBAR register) for stage-2
context banks. The VMID is valid and often used in stage-1 context banks
as well. Program it in both cases.
Change-Id: Id4381e67f1c4c318c3137a5d56256af711491f08
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
dev_err already prints the device name. Remove the `dev_name(dev)' from
the print statement in arm_smmu_iova_to_phys_hard.
Change-Id: I5f5bcaf57349aba157789070e2f1441184ee62a6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently there could be races between different domains performing an
ATOS operation. Fix this by taking the domain lock around the ATOS.
Change-Id: I32e6e0a0c67ce82aff5abfc892098c684d399b2d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, we return garbage when iova_to_phys fails when we're using
hardware translation features (ATOS). iova_to_phys should return 0 when
it fails. Fix this.
Change-Id: I3bfefce7315e43a684c7c29c4fa7ef5960182ea8
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ARM SMMUs support memory access control via some bits in the translation
table descriptor memory attributes. Currently we assume all translations
are "unprivileged". Add support for privileged mappings, controlled by
the IOMMU_PRIV prot flag.
Also sneak in a whitespace change for consistency with nearby code.
Change-Id: I9d922c19d7dc8e64e07b216fc1bab373aa2e733d
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some platforms require certain implementation-defined registers to be
programmed when first attaching to the SMMU. Add support for this via
specifying register offset, value pairs in the device tree.
Change-Id: Iac2fe42684c3849a24d0d1251a206954262257c5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Work around a hardware bug where some SMMUs cache invalid PTEs and mess
up the transaction logic. PTEs must be zero'd out completely rather than
just setting the last two bits to zero to work around this bug.
Change-Id: I98ac18142eda6f7817c78ab27a4850cdecf24913
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We should avoid setting the ARM SMMU driver up as the IOMMU for the PCI
bus if there aren't any IOMMUs on the bus. Provide a Kconfig knob to
control this.
Change-Id: I748f689cd5f512aa7e45e146c6d7ddccadc47e44
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
The ACTLR must be programmed according to the hardware design in order
for the coherent table walk feature to work on certain Qualcomm
hardware. Provide a new compatible string ("qcom,smmu-v2") to indicate
the relevant hardware and do the programming as needed.
Change-Id: I7e807384c821fc3d07274f35726abb28d0d75ee0
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Under certain conditions coherent hardware translation table walks can
result in degraded performance. Add a new domain attribute to
disable/enable this feature in generic code along with the domain
attribute setter and getter to handle it in the ARM SMMU driver.
Change-Id: Ic9812d5047af19093ec2db8246a7f0f9ccb626ab
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Add a workaround for some buggy hardware that requires a TLB invalidate
operation to occur at map time. Activate the feature with the
qcom,smmu-invalidate-on-map boolean DT property.
Change-Id: I081a279fead983ae3d736b44cda371078e55a750
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Generic IOMMU device tree bindings were recently added in [bb1e25b9ed:
"devicetree: Add generic IOMMU device tree bindings"]. Implement the
bindings in the ARM SMMU driver.
See Documentation/devicetree/bindings/iommu/iommu.txt for the bindings
themselves.
Change-Id: I318b4cc347caccf3e142ce021057599eaea07814
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, we provide the iommu_ops.iova_to_phys service by doing a
table walk in software to translate IO virtual addresses to physical
addresses. On SMMUs that support it, it can be useful to ask the SMMU
itself to do the translation. This can be used to warm the TLBs for an
SMMU. It can also be useful for testing and hardware validation.
Since the address translation registers are optional on SMMUv2, only
enable hardware translations when using SMMUv1 or when SMMU_IDR0.S1TS=1
and SMMU_IDR0.ATOSNS=0, as described in the ARM SMMU v1-v2 spec.
Change-Id: I07157ced4b2d8a5974b69f84f0ba47ca1b99481b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
