This reverts commit fd3161483bd7af420b503a8e63d3c1f24a7cf936. This code
is about to be ripped out in favor of the generic io-pgtable code, so
leaving it in will only cause merged conflicts.
Change-Id: I1404f79f08340d3de4a52155c3c82fae6df6515c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit 0385f613fc41c394873d64785c073dffdfaf0fc4. This code
is about to be ripped out in favor of the generic io-pgtable code, so
leaving it in will only cause merged conflicts.
Change-Id: Id8474f2a400736c288d9db5f9608eddbd24b0793
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit 433a4b6a37c8f96295a7e80cb603c2318872c1f2. An
equivalent patch is being brought in with the upcoming page table
refactoring, so rip this guy out in preparation.
This also reverts commit af3e3c36d08077856640b28d68c9f99d2188a000, which
introduced users of the IOMMU_NOEXEC flag.
Change-Id: Ibb36f60f2148170d8a01632043bf36b5f8c60777
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Following commit implements mapping of sg virtually contiguously.
"arm64: dma-mapping: map sg lists into the SMMU as virtually
contiguous"
a03f74ef16cc73531795176d3ea8b82b66ed0146
This has been left for ARM (32-bit). Implement the same for ARM.
Change-Id: Ibf67f29a60b8d19e526c4719590f2f473ea9dca5
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
__hvc should take 11 arguments instead of 10. Add the missing x7
argument to avoid compilation error if the driver gets enabled on
32bit targets.
Change-Id: I09985235fdbfb64e81743a71ceb2764c32d3a3b1
Signed-off-by: Pratik Patel <pratikp@codeaurora.org>
Add API support for calling into the hypervisor. This will allow
various drivers to avail hypervisor services.
Change-Id: I0a0e8f8fe13a550ad20c5421b712e207933c82f3
Signed-off-by: Pratik Patel <pratikp@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Usually when an SMMU probes we do a sanity check on the SMR registers to
make sure they can fully support all of the mask bits. This check can
cause problems for use cases where the SMMU is already in use when the
SMMU probes. For example, for continuous splash screen support, the
stream matching table is programmed before control is even turned over
to Linux.
Add an option to skip this sanity check.
Change-Id: I51a9231fcd8b73034f1a1ca69e4fbb7e632635fa
Signed-off-by: Dhaval Patel <pdhaval@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
PCI-e devices on MSM systems need to have their stream IDs configured at
device add time, since they're not known at system design time and
therefore can't be placed in the device tree. Add the necessary calls
into the MSM PCI-e driver to obtain stream IDs for devices behind PCI-e
at device add time.
Change-Id: I3645a525c3ab5ef6d89eeaa99894542bd3aa261f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The logic for the iommu executable flag is inverted now and
all the iommu mappings are executable by default.
Provide the IOMMU_NOEXEC flag where the mapping needs to be non-executable.
Change-Id: Ifa0aa3d17ae79c16abdf66d2177a09b868a9f45f
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
[pdaly@codeaurora.org Remove kgsl/display modifications]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Exposing the XN flag of the SMMU driver as IOMMU_NOEXEC instead of
IOMMU_EXEC makes it enforceable, since for IOMMUs that don't support
the XN flag pages will always be executable.
Change-Id: Ib785acd8a188fa95aea9991116139a392862764e
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: a720b41c41
[rvaswani@codeaurora.org: resolve trivial merge conflicts]
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
The arm_smmu_map operation needs to be in an atomic
context to accomodate certain IPA usecases which take
place in atomic context.
Change-Id: I9049c43167bcc6d1140f6154d17733345b415d7b
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
This flag is used for specifying access to device memory. SMMU would apply
device memory attributes for a DMA transaction. This is required for setting
access to GIC registers, for generating message interrupts. This would ensure that
transactions targetting device memory are not gathered or reordered.
CRs-Fixed: 792402
Change-Id: Ief623e16e4f2dfc99ef6745459777269f1ab7ac6
Signed-off-by: Varun Sethi <Varun.Sethi@freescale.com>
Patch-mainline: iommu @ 10/06/14 @ 10:28
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Currently we program all stream IDs based on the configuration specified
by clients in their device tree nodes, so we've always required any
`iommus' entries to have *exactly* 1 stream ID per list element.
However, there are some devices that might not have statically-defined
SIDs that can be placed in the device tree (devices behind PCI-e, for
example). Add support for this by accepting stream-ID-less `iommus'
entries in the device tree.
Change-Id: I62745dc4159114181fa51b22c732935cb47f60bc
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This is used for indicating device memory type for a DMA transaction. IOMMU
driver would set up attributes indicationg access to device memory.
CRs-Fixed: 792402
Change-Id: Ifd6f2288353ef737ff4c15d9bd1514c66af8fdd2
Signed-off-by: Varun Sethi <Varun.Sethi@freescale.com>
[neetid: changed IOMMU_DEVICE to be on bit 5 instead of bit 4]
Patch-mainline: iommu @ 10/06/14, 10:28
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
For secure domains, the page tables need to be assigned
to the correct VMIDs. Add support for the same.
Change-Id: Ic46e61300463d969b35ebc5d4d20cfc30a4ba476
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
IOMMU virtual addresses use the dma_addr_t type since they can be up to
64-bits. We're currently using an `unsigned int' to store our IOVA in
arm_iommu_unmap_sg, which could result in truncation. Use the correct
type for an I/O virtual address: dma_addr_t.
This was previously fixed for arm_iommu_map_sg in
[02454d7f9feeb: "arm64: dma-mapping: use correct type for iova"].
Make the same fix in arm_iommu_unmap_sg.
Change-Id: Ib22a9600f33e6fa155812b08d67d62f72af0ad8e
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
We're currently asking the IOMMU layer to do an iova-to-phys translation
in .unmap_page and .sync_single_for_* in the IOMMU DMA mapper. This can
be a costly operation since it will need to walk the domain's page
tables, either in software or in hardware. Also, in some
less-than-ideal implementations of iommu_iova_to_phys this might
actually involve sleeping operations.
Avoid this overhead by saving the physical address of the buffer in the
dma_iommu_mapping structure in .map_page, using it later instead of
iommu_iova_to_phys.
Change-Id: Ic53b91a222dab01cfcdc34246a847a8c399adfb6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Now that the non-consistent behavior is fully integrated into the
dma mapping subsystem, allow user space to mmap nonconsistent dma
buffers.
Change-Id: I1b95ba9a1751e00a5812d9df582af20f316fcafe
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Conflicts:
arch/arm64/include/asm/dma-mapping.h
An extra check was added in [3c8766d0c: "iommu/arm-smmu: Do not access
non-existing S2CR registers"] to make sure we didn't add stream IDs
greater than the maximum allowed value when stream matching is not
supported. That check was designed to happen at device tree parse time,
after probing the SMMU to get the maximum allowed SID value. However,
we've added support for clocks since that time, which are also parsed
from the device tree. So we can't probe the device to see what features
it supports until we parse the device tree so that we can enable clocks.
Fix this catch-22 by dropping the max stream ID value check. This won't
affect current MSM targets at all since they all support the stream
matching feature, so the sanity check was never applicable.
An alternate solution might be to do a multi-phase parse of the device
tree to pull out just the necessary configuration to power on the
SMMU (clocks, etc), then probe the device for features, then parse the
rest of the device tree (during which time we'd do this sanity check).
Change-Id: I679bbde96a4b8800da0c6d7a5a186d0fe7bd0d75
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ARM64's dma-mapping.c is getting support for IOMMUs. Select
ARM64_DMA_USE_IOMMU for ARM64 so that the generic DMA APIs can leverage
IOMMU mappings.
Change-Id: I5adbac5073b6669f6f53680b8d57b7f8eeccc016
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The real arm_iommu_create_mapping function returns an ERR_PTR (not NULL) on
failure. Make the stub version match that convention.
Change-Id: I1df954ee5b9037778f27fba2e626621740abf782
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This attribute can be used to get the base address of the page tables
for the given domain. This can be useful for some drivers that switch
page tables around out-of-band (for example, per-process page tables
managed by the GPU).
Change-Id: I7c67c4c6435e5aadba129a53be532c2396dccf9c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Add the IOMMU_PRIV attribute, which is used to indicate privileged
mappings.
Change-Id: If8576c09898bd7953475ab72e178d3288a763b0c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit b20594eac137b78c6e55eb1872ceb72ea98edbef, whose
original intent was to workaround a bug in the PCI driver that has since
been fixed. Also rip it out of msm_defconfig.
Change-Id: Ied8349545f872b4fc23b46b15d13677279ad9ae1
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Clients are allowed to attach/detach and map/unmap independently. I.e.,
they don't have to be attached at the time they map or unmap. Currently
if they detach and unmap at the same time there's a race condition where
the SMMU associated with the domain could go away while we try to use
it. Fix this by locking the domain lock everywhere that domain->smmu is
used.
Change-Id: I2b74c0863c28e3bb87e8bd45dae363c8e67e008b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We'd like to use the domain lock to protect against a race condition
where the domain's device pointer is used after the domain has been
detached from the device. In order to prepare for this, change the
domain lock to a mutex, since we'll be holding during times where we
need to do non-atomic operations (like enabling clocks).
Change-Id: Ib57812851487c0f0c4833c65f0b1c63e010385bf
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There are certain use cases that require the stream matching table to be
programmed without actually enabling translations on the
SMMU (i.e. leaving SCR0.M=0). For example, when a hypervisor is
controlling the stage-2 context bank of a nested configuration where
stage-1 needs to be bypassed. This mode of operation is described in
the ARM SMMU spec as "stage 1 and stage 2 contexts are valid, but the
SMMU is not enabled for stage 1 translation" (Section 2.1: "Overview of
SMMU operation").
The easiest way to get the stream-matching table programmed correctly is
to program it as usual from Linux but just leave SCR0.M=0. Add a DT
option to do this.
Change-Id: I065a38f845ae8873bc51221fe64a39b1908032d6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Thulium v1 has an ATOS hardware errata that requires that we call into
TZ to do some fixups. Add a DT option to enable this workaround.
Change-Id: Ida2fecf1b40ba0f37c9cacc4296b0e8e46db071c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some targets require calling into TZ for various bits of SMMU
configuration. Add a library to provide a clean interface for such
configuration.
Change-Id: I1dc5cd21d4a09e321039d69cc760eaf6f356c6cf
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Because of a hardware errata, arm_smmu_iova_to_phys_hard will soon be
making a TZ call to do a workaround. However, we're currently using a
spinlock to ensure atomicity of ATOS due to another hardware errata, but
scm_call is a sleeping function, so this results in a sleeping BUG. Fix
this by making the atos lock a mutex instead of a spinlock.
This isn't exactly correct since iommu_iova_to_phys itself might be
called from atomic context, so we really shouldn't be taking a mutex.
However, we don't seem to have any use cases where it will be called
from atomic context, but we should revert all of this and go back to a
spinlock as soon as this hardware errata goes away (which will happen
when Thulium v1 dies).
Change-Id: I61ea37bb3e6989fe5db43c4e828fc6473885db1e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We're currently mapping a page in arm_smmu_flush_pgtable without ever
unmapping it. Fix this by calling dma_unmap_page on the returned dma
address. Since the only reason we're calling dma_map_page is to make
sure it actually gets flushed out to RAM, we can just call
dma_unmap_page immediately following the map.
Without this, eventually swiotlb runs out of memory and starts printing
things like:
arm-smmu d00000.arm,smmu: swiotlb buffer is full (sz: 128 bytes)
Change-Id: I69421de6e2189a16cd88f225e2698cd88d669dff
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
To aide in debugging, print the value of the IDR0 register when no
translation support is found, which would generally indicate a hardware
bug or missing clocks/power.
Change-Id: I22dd44fc30012c72a7d04088ab079076b6837f7e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Since we use dma_map_page() as an architecture-independent means of
making page table updates visible to non-coherent SMMUs, we need to
have a suitable DMA mask set to discourage the DMA mapping layer from
creating bounce buffers and flushing those instead, if said page tables
happen to lie outside the default 32-bit mask.
Change-Id: Ic5accfc4da5dff76123d7c524a9da77c9d88a847
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Patch-mainline: iommu @ 5 Mar 2015 18:56:36 +0000
[mitchelh@codeaurora.org: context differences]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Thulium v1 has a hardware bug that requires us to zero out page table
entries completely rather than just setting the valid bits to 0. The
previous workaround for this [165066b85d: "iommu/arm-smmu: work around
transaction hardware bug"] was actually not quite right since it still
allowed the page frame number to be set in the page table entry. Fix
this by actually zero'ing out the pte.
Change-Id: I37537874ea4c5d1c00db7de65f70edd93ce63b90
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Thulium has a hardware errata that requires that we avoid using ASID 0.
Implement the workaround in the SMMU driver.
Change-Id: Ia44ab1b385450994641c5f5ccb3dba4a462a8033
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Mapping and unmapping are more often than not in the critical path.
map_sg allows IOMMU driver implementations to optimize the process
of mapping buffers into the IOMMU page tables.
Instead of mapping a buffer one page at a time and requiring potentially
expensive TLB operations for each page, this function allows the driver
to map all pages in one go and defer TLB maintenance until after all
pages have been mapped.
Additionally, the mapping operation would be faster in general since
clients does not have to keep calling map API over and over again for
each physically contiguous chunk of memory that needs to be mapped to a
virtually contiguous region.
Change-Id: I1f3dd2c3cf67b3db40ee1793580d6af5fec1247d
Signed-off-by: Olav Haugan <ohaugan@codeaurora.org>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Git-commit: 315786ebbf
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[mitchelh: fix existing callers and implementations of
iommu_{map,unmap}_range to match the new function names and APIs,
maintaining stubs for the old API so that out-of-tree modules can
continue to compile]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We're currently wiping out the `far' variable in our context fault
handler when CONFIG_64BIT is enabled. Fix the bug.
Change-Id: I86e57c7de8bfc10118f902d6702d3b9059a8762a
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, we fall back to a software table walk if ATOS fails. This
can mask problems that might be harder to debug later. Rather than
doing a software table walk and returning success, just print the result
of the software table walk to the log and return failure.
Change-Id: I28335b33977e60aea43d2c0ee1f4571a14a5e191
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we don't print a whole lot in the context fault handler. Make
it more verbose to facilitate debugging.
Change-Id: I4c0570184e0129c1d0c40f5d632a412de7c81cd4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We need to make sure the TLBIALL is actually done before proceeding with
the ATOS since the ATOS won't succeed if the TLB hasn't been
invalidated. Add a TLBSYNC to ensure that it finishes.
Change-Id: I55836039d7ded5ba8889bc81ee474624e98ef491
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We need to do some sleeping operations (like enabling clocks) in the
global fault handler. Move to a threaded handler to avoid BUGs due to
sleeping in an atomic context.
Change-Id: I88eed743bfad4cc33daf3b9f81255587c72b0167
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some hardware requires special fixups to recover from address size
faults. Rather than applying the fixups add an option to just BUG since
address size faults are due to a fundamental programming error from
which we don't care about recovering anyways.
Change-Id: Ibb70e4ec00683562dae9f3239b286daa5deabd45
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There is a hardware errata that could result in bus hang during context
fault processing. The work around given by the hardware team is to
issue a TLBSYNC and terminate the transaction. Implement the
workaround and provide a DT option to activate it.
Change-Id: Iede30ff68676188af1249fd2fd776bc84f224dc6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, we program the TCR2.SEP to be equal to the input address
size. This makes it impossible to use the top bit of the input address
space since that causes TTBR1 to be selected, which we don't program or
use. Since we don't even use sign-extended addresses, fix this by
always using the "no-sign" value for TCR2.SEP as specified by the ARM
SMMUv2 spec.
Change-Id: I9155b62fe59c66185c6b9fba7f8bba8798b45785
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we're setting page table cacheability to "inner shareable"
when coherent table walking is enabled, but it should actually be "outer
shareable". Fix this.
Change-Id: I7939bb75da8327f01a2fcd941a0d8358de6f0bc5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently when an SMMU client accesses an unmapped address the SMMU does
a bypass and propagates the request. This can make it difficult to
debug problems when clients are accessing bogus addresses. Enable the
sCR0 bit that causes the SMMU to raise an unidentified stream fault
rather than falling back to bypass when no mapping is found.
Change-Id: Ia0e40010c79b8a75954ae13ae60aca2d82ff76b6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
On some targets, context banks can be dedicated for use by other
execution environments outside of Linux. However, we currently assume
that we own all of the context banks during SMMU initialization in
arm_smmu_device_reset. This can mess up the other execution
environments since we are trampling on their SCTLR, SMR, and S2CR.
Provide a DT option to skip this initialization altogether, since the
other execution environment should have already initialized the SMMU
anyways.
Change-Id: I0ed4cbbcdad596f9201f83cb7d0e28a289e18a6e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Context interrupts can call domain-specific handlers which might sleep.
Currently we register our handler with request_irq, so our handler is
called in atomic context, so domain handlers that sleep result in an
invalid context BUG. Fix this by using request_threaded_irq.
This also prepares the way for doing things like enabling clocks within
our interrupt handler.
Change-Id: I2f12f2aa234b62532807a208cd70f2d705083343
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some hardware is capable of retaining register values during power
collapse. Add an option (configurable via DT) to enable this feature.
This is implemented by always enabling/disabling regulators every time
clocks are enabled/disabled.
Change-Id: I89d9a4f4a2eb29f0868b309d55a77cc4ed50e22e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Hardware requires that the ATOS command be issued with a single 64-bit
write instead of two 32-bit writes as we're currently doing. Fix this.
Change-Id: I43104c89c2f27b75d1176c8cbcd214666321244f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
