Currently we walk each last-level leaf pte during unmap and zero them
out individually. Since these last-level ptes are all contiguous (up to
512 entries), optimize the unmapping process by simply zeroing them all
out at once rather than operating on them individually.
Change-Id: I21d490e8a94355df4d4caecab33774b5f8ecf3ca
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We need to disable clocks and unlock a mutex if
arm_smmu_init_domain_context fails during attach. Add the proper
cleanup.
Change-Id: Id097ee6815d63b61dd3db1843f054ed991d34224
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Rather than calling the tlb maintenance routines throughout the course
of the unmap operation, just flush the entire tlb for the context in
question all at once, at the very end of the unmap. This greatly
improves performance for large page tables (which is common for large
buffers in a heavily fragmented system).
In my testing, this optimization gave a ~10% speedup when unmapping 64K.
Change-Id: Iaa2b211e730dad6bd9235ef98dd2a89cf541e663
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The ARM LPAE io-pgtable provider now has support for .map_sg. Wire it
up.
Change-Id: I2eaea6f245e1059582f3bf04829e9e3f24675782
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
io-pgtable-arm has just gotten support for .map_sg. Add a test to the
suite of self-tests for this.
Change-Id: Iba56bb801c1f9ef151827598022411c95d389faa
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Mapping an entire scatterlist at once is faster than calling iommu_map
on each link individually. Implement .map_sg in the ARM LPAE page table
allocator so that drivers using the allocator can leverage this
performance boost.
Change-Id: I77f62a2566058693c3f58fc0b05d715a780ae5d8
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The default implementation of map_sg (provided by default_iommu_map_sg
from the IOMMU API layer) actually just calls iommu_map repeatedly.
Often times there are locks or other initial setup in the implementation
of iommu_map, so there's opportunity to drastically improve performance
by factoring that out of the loop. For this reason, io-pgtable
providers might want to implement .map_sg. Add the declaration to the
io_pgtable_ops structure.
Change-Id: I5aa5e4e24a68e303ce2c005dc1dd0b33d5635ab3
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful in IOMMU drivers. Export it.
Change-Id: I4c423d256312250f1e33ca8d64dfe1626f008b5e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Previously we were relying on the callers of the tlb invalidation
functions to enable clocks. However, the new io-pgtable framework
doesn't know anything about clocks and calls directly into the tlb
invalidation functions, so we need to put the clocks enable/disable
inside the tlb functions themselves.
Change-Id: I73a55e3f78deb3501df5615e22cd298fba6e4551
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The newly-introduced smmu_domain.init_mutex can be used everywhere that
we're currently using smmu_domain.lock to protect smmu_domain.smmu.
Move to `init_mutex' and delete `lock'.
Change-Id: I92aa237e72cafd0c8fa39fe6279c44bd194c11b4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some clients need to get the base address of the page table for a given
IOMMU domain. This functionality is provided by the
DOMAIN_ATTR_PT_BASE_ADDR domain attribute. Implement it in the ARM SMMU
driver by returning ttbr[0] from the page table config.
Change-Id: Ie9b0241c7c7df18b25761bae79c1be1a283ff3a4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The pgtbl_cfg object has a few handy properties that we'd like to make
use of later (returning the pgd in a domain attribute, for example).
Keep track of the domain pgtbl_cfg in the domain structure.
Change-Id: Icce9638a270ca98d6ed6d61b3ef1c35d42a869dc
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The quirk causes the Non-Secure bit to be set in all page table entries.
Change-Id: I937fb7dec4214eca33f8014c664cfc5c99cb0027
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: c896c132b0
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The ARM SMMU can walk LPAE page tables, so make use of the generic
allocator.
Change-Id: I7acd0e6c6e8046a4f0497a8fbdba46986192f83f
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: 518f713624
[mitchelh: merge with our changes to domain locking, non-shareable
page tables, and pte zero'ing, some of which will need to be added
back now in the generic allocator code]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This patch adds a series of basic self-consistency tests to the ARM LPAE
IO page table allocator that exercise corner cases in map/unmap, as well
as testing all valid configurations of pagesize, ias and stage.
Change-Id: I703df977b7e5914e0ccf9aaca2174cf5956dd604
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: fe4b991dcd
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
A number of IOMMUs found in ARM SoCs can walk architecture-compatible
page tables.
This patch adds a generic allocator for Stage-1 and Stage-2 v7/v8
long-descriptor page tables. 4k, 16k and 64k pages are supported, with
up to 4-levels of walk to cover a 48-bit address space.
Change-Id: I32740cfa795c55e0d3683b42105b4f49c9dcf984
Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: e1d3c0fd70
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This patch introduces a generic framework for allocating page tables for
an IOMMU. There are a number of reasons we want to do this:
- It avoids duplication of complex table management code in IOMMU
drivers that use the same page table format
- It removes any coupling with the CPU table format (and even the
architecture!)
- It defines an API for IOMMU TLB maintenance
Change-Id: Ia7c22397f0afe94898cd91d8d9c30be985993d1d
Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: fdb1d7be7c
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit c8690d83e1e1b09d55ed75d6776efe42915b9c95 since it
won't compile when arm-smmu.c moves to the generic page table
allocator (since cfg.pgd is being deleted there). We'll bring this back
in with the appropriate fix once the generic page table allocator patch
is in.
Change-Id: I956d902999a05653e70f7cb7a90ac5010f9926de
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
When domains are set with the DOMAIN_ATTR_NESTING flag, we must ensure
that we allocate them to stage-2 context banks if the hardware permits
it.
This patch adds support for the attribute to the ARM SMMU driver, with
the actual stage being determined depending on the features supported
by the hardware.
Change-Id: I17e027388efbeaae65886160f35a5f8068fd7734
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: c752ce45b2
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[mitchelh: resolved conflicts with our existing domain attr
setter/getter, and locking differences]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently the driver registers IOMMU bus operations for all busses even
if no ARM SMMU is present on a system. Depending on the driver probing
order this prevents the driver for the real IOMMU to register itself as
the bus-wide IOMMU.
Change-Id: I345b0e922ae3d0d71622217fd869747d25ae2bb4
Signed-off-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: 0e7d37adbe
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Exposing the XN flag of the SMMU driver as IOMMU_NOEXEC instead of
IOMMU_EXEC makes it enforceable, since for IOMMUs that don't support
the XN flag pages will always be executable.
Change-Id: Ic6b1f0ae8873035dcb798e9c996396fb2bc4921e
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: a720b41c41
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[mitchelh: context fixups]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
In ["iommu/arm-smmu: fix architecture version detection"] we added the
ability to determine the SMMU architecture version from DT match data by
using of_match_node, but neglected to check the return value for NULL.
Fix this.
Change-Id: Ib9049f2b2f1204b52bc07ba8caf075449af4cd35
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit e6e701ddb09b2b6b43ffc7ff788701eda03703b8. This code
is about to be ripped out in favor of the generic io-pgtable code, so
leaving it in will only cause merged conflicts.
Don't revert the introduction of the DOMAIN_ATTR_SECURE_VMID attribute
to avoid unnecessary churn in client code.
Change-Id: I58af07686f11b14a82d44bd7999ba50127530f1f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit fd3161483bd7af420b503a8e63d3c1f24a7cf936. This code
is about to be ripped out in favor of the generic io-pgtable code, so
leaving it in will only cause merged conflicts.
Change-Id: I1404f79f08340d3de4a52155c3c82fae6df6515c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit 0385f613fc41c394873d64785c073dffdfaf0fc4. This code
is about to be ripped out in favor of the generic io-pgtable code, so
leaving it in will only cause merged conflicts.
Change-Id: Id8474f2a400736c288d9db5f9608eddbd24b0793
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit 433a4b6a37c8f96295a7e80cb603c2318872c1f2. An
equivalent patch is being brought in with the upcoming page table
refactoring, so rip this guy out in preparation.
This also reverts commit af3e3c36d08077856640b28d68c9f99d2188a000, which
introduced users of the IOMMU_NOEXEC flag.
Change-Id: Ibb36f60f2148170d8a01632043bf36b5f8c60777
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Following commit implements mapping of sg virtually contiguously.
"arm64: dma-mapping: map sg lists into the SMMU as virtually
contiguous"
a03f74ef16cc73531795176d3ea8b82b66ed0146
This has been left for ARM (32-bit). Implement the same for ARM.
Change-Id: Ibf67f29a60b8d19e526c4719590f2f473ea9dca5
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
__hvc should take 11 arguments instead of 10. Add the missing x7
argument to avoid compilation error if the driver gets enabled on
32bit targets.
Change-Id: I09985235fdbfb64e81743a71ceb2764c32d3a3b1
Signed-off-by: Pratik Patel <pratikp@codeaurora.org>
Add API support for calling into the hypervisor. This will allow
various drivers to avail hypervisor services.
Change-Id: I0a0e8f8fe13a550ad20c5421b712e207933c82f3
Signed-off-by: Pratik Patel <pratikp@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Usually when an SMMU probes we do a sanity check on the SMR registers to
make sure they can fully support all of the mask bits. This check can
cause problems for use cases where the SMMU is already in use when the
SMMU probes. For example, for continuous splash screen support, the
stream matching table is programmed before control is even turned over
to Linux.
Add an option to skip this sanity check.
Change-Id: I51a9231fcd8b73034f1a1ca69e4fbb7e632635fa
Signed-off-by: Dhaval Patel <pdhaval@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
PCI-e devices on MSM systems need to have their stream IDs configured at
device add time, since they're not known at system design time and
therefore can't be placed in the device tree. Add the necessary calls
into the MSM PCI-e driver to obtain stream IDs for devices behind PCI-e
at device add time.
Change-Id: I3645a525c3ab5ef6d89eeaa99894542bd3aa261f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The logic for the iommu executable flag is inverted now and
all the iommu mappings are executable by default.
Provide the IOMMU_NOEXEC flag where the mapping needs to be non-executable.
Change-Id: Ifa0aa3d17ae79c16abdf66d2177a09b868a9f45f
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
[pdaly@codeaurora.org Remove kgsl/display modifications]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Exposing the XN flag of the SMMU driver as IOMMU_NOEXEC instead of
IOMMU_EXEC makes it enforceable, since for IOMMUs that don't support
the XN flag pages will always be executable.
Change-Id: Ib785acd8a188fa95aea9991116139a392862764e
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: a720b41c41
[rvaswani@codeaurora.org: resolve trivial merge conflicts]
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
The arm_smmu_map operation needs to be in an atomic
context to accomodate certain IPA usecases which take
place in atomic context.
Change-Id: I9049c43167bcc6d1140f6154d17733345b415d7b
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
This flag is used for specifying access to device memory. SMMU would apply
device memory attributes for a DMA transaction. This is required for setting
access to GIC registers, for generating message interrupts. This would ensure that
transactions targetting device memory are not gathered or reordered.
CRs-Fixed: 792402
Change-Id: Ief623e16e4f2dfc99ef6745459777269f1ab7ac6
Signed-off-by: Varun Sethi <Varun.Sethi@freescale.com>
Patch-mainline: iommu @ 10/06/14 @ 10:28
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Currently we program all stream IDs based on the configuration specified
by clients in their device tree nodes, so we've always required any
`iommus' entries to have *exactly* 1 stream ID per list element.
However, there are some devices that might not have statically-defined
SIDs that can be placed in the device tree (devices behind PCI-e, for
example). Add support for this by accepting stream-ID-less `iommus'
entries in the device tree.
Change-Id: I62745dc4159114181fa51b22c732935cb47f60bc
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This is used for indicating device memory type for a DMA transaction. IOMMU
driver would set up attributes indicationg access to device memory.
CRs-Fixed: 792402
Change-Id: Ifd6f2288353ef737ff4c15d9bd1514c66af8fdd2
Signed-off-by: Varun Sethi <Varun.Sethi@freescale.com>
[neetid: changed IOMMU_DEVICE to be on bit 5 instead of bit 4]
Patch-mainline: iommu @ 10/06/14, 10:28
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
For secure domains, the page tables need to be assigned
to the correct VMIDs. Add support for the same.
Change-Id: Ic46e61300463d969b35ebc5d4d20cfc30a4ba476
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
IOMMU virtual addresses use the dma_addr_t type since they can be up to
64-bits. We're currently using an `unsigned int' to store our IOVA in
arm_iommu_unmap_sg, which could result in truncation. Use the correct
type for an I/O virtual address: dma_addr_t.
This was previously fixed for arm_iommu_map_sg in
[02454d7f9feeb: "arm64: dma-mapping: use correct type for iova"].
Make the same fix in arm_iommu_unmap_sg.
Change-Id: Ib22a9600f33e6fa155812b08d67d62f72af0ad8e
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
We're currently asking the IOMMU layer to do an iova-to-phys translation
in .unmap_page and .sync_single_for_* in the IOMMU DMA mapper. This can
be a costly operation since it will need to walk the domain's page
tables, either in software or in hardware. Also, in some
less-than-ideal implementations of iommu_iova_to_phys this might
actually involve sleeping operations.
Avoid this overhead by saving the physical address of the buffer in the
dma_iommu_mapping structure in .map_page, using it later instead of
iommu_iova_to_phys.
Change-Id: Ic53b91a222dab01cfcdc34246a847a8c399adfb6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Now that the non-consistent behavior is fully integrated into the
dma mapping subsystem, allow user space to mmap nonconsistent dma
buffers.
Change-Id: I1b95ba9a1751e00a5812d9df582af20f316fcafe
Signed-off-by: Laura Abbott <lauraa@codeaurora.org>
Conflicts:
arch/arm64/include/asm/dma-mapping.h
An extra check was added in [3c8766d0c: "iommu/arm-smmu: Do not access
non-existing S2CR registers"] to make sure we didn't add stream IDs
greater than the maximum allowed value when stream matching is not
supported. That check was designed to happen at device tree parse time,
after probing the SMMU to get the maximum allowed SID value. However,
we've added support for clocks since that time, which are also parsed
from the device tree. So we can't probe the device to see what features
it supports until we parse the device tree so that we can enable clocks.
Fix this catch-22 by dropping the max stream ID value check. This won't
affect current MSM targets at all since they all support the stream
matching feature, so the sanity check was never applicable.
An alternate solution might be to do a multi-phase parse of the device
tree to pull out just the necessary configuration to power on the
SMMU (clocks, etc), then probe the device for features, then parse the
rest of the device tree (during which time we'd do this sanity check).
Change-Id: I679bbde96a4b8800da0c6d7a5a186d0fe7bd0d75
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
ARM64's dma-mapping.c is getting support for IOMMUs. Select
ARM64_DMA_USE_IOMMU for ARM64 so that the generic DMA APIs can leverage
IOMMU mappings.
Change-Id: I5adbac5073b6669f6f53680b8d57b7f8eeccc016
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The real arm_iommu_create_mapping function returns an ERR_PTR (not NULL) on
failure. Make the stub version match that convention.
Change-Id: I1df954ee5b9037778f27fba2e626621740abf782
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This attribute can be used to get the base address of the page tables
for the given domain. This can be useful for some drivers that switch
page tables around out-of-band (for example, per-process page tables
managed by the GPU).
Change-Id: I7c67c4c6435e5aadba129a53be532c2396dccf9c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Add the IOMMU_PRIV attribute, which is used to indicate privileged
mappings.
Change-Id: If8576c09898bd7953475ab72e178d3288a763b0c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit b20594eac137b78c6e55eb1872ceb72ea98edbef, whose
original intent was to workaround a bug in the PCI driver that has since
been fixed. Also rip it out of msm_defconfig.
Change-Id: Ied8349545f872b4fc23b46b15d13677279ad9ae1
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Clients are allowed to attach/detach and map/unmap independently. I.e.,
they don't have to be attached at the time they map or unmap. Currently
if they detach and unmap at the same time there's a race condition where
the SMMU associated with the domain could go away while we try to use
it. Fix this by locking the domain lock everywhere that domain->smmu is
used.
Change-Id: I2b74c0863c28e3bb87e8bd45dae363c8e67e008b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We'd like to use the domain lock to protect against a race condition
where the domain's device pointer is used after the domain has been
detached from the device. In order to prepare for this, change the
domain lock to a mutex, since we'll be holding during times where we
need to do non-atomic operations (like enabling clocks).
Change-Id: Ib57812851487c0f0c4833c65f0b1c63e010385bf
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There are certain use cases that require the stream matching table to be
programmed without actually enabling translations on the
SMMU (i.e. leaving SCR0.M=0). For example, when a hypervisor is
controlling the stage-2 context bank of a nested configuration where
stage-1 needs to be bypassed. This mode of operation is described in
the ARM SMMU spec as "stage 1 and stage 2 contexts are valid, but the
SMMU is not enabled for stage 1 translation" (Section 2.1: "Overview of
SMMU operation").
The easiest way to get the stream-matching table programmed correctly is
to program it as usual from Linux but just leave SCR0.M=0. Add a DT
option to do this.
Change-Id: I065a38f845ae8873bc51221fe64a39b1908032d6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
