In atomic context, gen_pool_alloc allocates a single page large
enough to accomodate the requested size. However __iommu_create_mapping
always maps pages assuming they are of size 4K. Thus only the first
4K of the buffer is mapped and a translation fault is generated
during an unmap.
Fix this by splitting the larger pages into 4K pages.
Change-Id: Ifcbe29477ad210204028486bd011470fe8b50852
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Keep upstream version of alloc_from_pool]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
The return value -EINVAL is returned in the case of invalid
arguments, and is not the correct value when the function is not
implemented.
Return -ENOSYS instead.
Change-Id: I196537f121d5a290fec74e2b7bcb1cfd490468c7
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Add ftrace start and end logging for map, iommu_map_sg and unmap
in order to facilitate performance testing.
Change-Id: I9ddf241ffa6cf519f6abece7b0820640f5ce1975
Signed-off-by: Liam Mark <lmark@codeaurora.org>
If something bad happens in arm_lpae_map_sg we try to clean up after
ourselves by unmapping any partial mappings that succeeded. However,
we're currently passing in the wrong iova to the unmap function during
the cleanup. Fix this.
Change-Id: Ieb30616141f3fb709d02abd147f9f598e2db07cc
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There appear to be bugs in the 4-level support of io-pgtable-arm. Work
around this by forcing all VAs to be 36 bits maximum so that we only
ever use 3 levels.
Change-Id: I5354afad05f74e12c51b86c97cdf1b2e86b68949
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Unmap returns a size_t all throughout the IOMMU framework. Make
io-pgtable match this convention.
Change-Id: Ice4c75a428f0f95a665e2fbe4210349d6f78c220
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Coherent table walk is broken on our system. Set page tables as outer
shareable so that the SMMU doesn't try to walk them in the cache.
Change-Id: Id9dd3d10139750b0dbb77842c12efd49e2672645
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit 0d02975d9ffd55f1c0fe5db08f45a9ee1d22f354 since it's
causing problems for some reason. This should really be debugged but
for now just revert it.
Change-Id: I31f382c1945cd1cd84dbbd3dfb715009b8442fe9
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
If dma_map_single() or dma_map_page() fail to create a mapping, clients
can check these errors by testing the returned address with
.dma_mapping_error(). The implementation was missing for DMA IOMMU
mapper. Add it.
Change-Id: I7e62bdbcd013cab915134fce9e55b1e7b580915a
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The dma_supported() API checks to see if the device can support DMA to
the memory described by the mask. Implement this for the DMA IOMMU
mapper by calling into the IOMMU's iommu_dma_supported API.
Change-Id: I5e31e386cd40c3f3312e1be1ca516c6e72213547
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The .dma_supported() checks to see if the device can support DMA to the
memory described by the mask. Implement this for the arm-smmu driver.
Change-Id: Ie494d574b613c1c76e6f878048c74444dc25902c
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The .dma_supported() checks to see if the device can support DMA to the
memory described by the mask. Add support for this operation in the
iommu layer
Change-Id: Icf37b9540aa68c2be3fd603a48402d6fcccd8208
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Certain clients need map and unmap operations to be in the atomic
context. This currently doesn't work since we need to prepare clocks
and enable regulators (for tlb invalidation), and take a mutex (to
protect the smmu_domain->smmu instance), which are all sleeping
operations.
Fix this by introducing a domain attribute to indicate that map and
unmap should be atomic. When this domain attribute is set we assume:
(1) clocks are already prepared
(2) regulators are already on
(3) the domain is attached to an SMMU
(1) and (2) allow us to skip preparing clocks and enabling regulators
during map and unmap and are taken care of internally in the
driver. (3) is a contract with clients that allows us to skip taking
the mutex that protects smmu_domain->smmu since we assume that the SMMU
instance is valid and will not be going away. If clients break this
contract then there are race conditions all over the place that will
eventually result in a NULL dereference (specifically, of
smmu_domain->smmu).
Change-Id: I3e21df02f7b0cd116fb558715eea16f2119f1d03
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There seems to be a bug in the unmap code that results in us leaving
stale mappings in the page table. We can actually live with this as
long as we invalidate the tlb when a new mapping comes in on the same
virtual address (to prevent the walker from using the old, bogus
iova->phys mappings).
Change-Id: If5923e853e7ec542b12ca954d5f1c22dec5e5bb2
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
With the recent page table refactor we've lost our code to assign SMMU
page tables to secure VMIDs. This functionality will be added back
later but for now just silently succeed when clients set this attribute.
This works for now because secure use cases haven't been brought up yet.
Change-Id: Ic2970e01f2899a9649c883f85aaaec828f2e7597
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We're currently BUG()'ing when we can't find a valid IOMMU page size
without printing any other information. Add some more information about
the parameters passed to the function to aide in debugging.
Change-Id: I1797bdfa2ef9d899ef4ffcb36fea769b67a1f991
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
In ARMv8, the output address from a page table walk is obtained by
combining some bits from the physical address in the leaf page table
entry with some bits from the input virtual address. The number of bits
that should be taken from the virtual address varies based on the lookup
level and descriptor type. However, we're currently always using
data->pg_shift bits, which is a constant.
Conveniently there's already a macro to compute the number of bits we
want (ARM_LPAE_LVL_SHIFT). Use this macro instead of data->pg_shift to
build the virtual address mask.
Change-Id: Id7f8aa2c553cc004e5d895d05c9226a896d22ce6
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The io-pgtable-arm unit tests currently only check a few mappings within
a range to determine whether or not that range "looks good". There can
be subtle bugs that don't show up until you get to a certain offset
within the range, etc. Check the entire range before assuming that it's
good.
Change-Id: I244a2150d38f57d95a5c81854cdeaf59ab4ace06
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently if we fail to allocate an IOVA from our VA pool we proceed to
map DMA_ERROR_CODE into the IOMMU, which is not what the client
requested or expected. Fix this by checking for an error return value
while allocating an IOVA and bailing out as appropriate.
Change-Id: I13df7b9ab52536b757d8ae0ca74a5b9e1e1438b5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Clients might want to map device memory into their SMMU. Add support
for these device mappings through the IOMMU_DEVICE flag.
Change-Id: I756720181aa0d531f4c56453ef832f81b36ffccd
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
If coherent hardware table walks are disabled via the
DOMAIN_ATTR_COHERENT_HTW_DISABLE attribute we need to flush the page
tables to memory every time we touch them. Do it.
Change-Id: I34189b1312eb294102eeaa59ad46d164777a6789
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we walk each last-level leaf pte during unmap and zero them
out individually. Since these last-level ptes are all contiguous (up to
512 entries), optimize the unmapping process by simply zeroing them all
out at once rather than operating on them individually.
Change-Id: I21d490e8a94355df4d4caecab33774b5f8ecf3ca
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We need to disable clocks and unlock a mutex if
arm_smmu_init_domain_context fails during attach. Add the proper
cleanup.
Change-Id: Id097ee6815d63b61dd3db1843f054ed991d34224
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Rather than calling the tlb maintenance routines throughout the course
of the unmap operation, just flush the entire tlb for the context in
question all at once, at the very end of the unmap. This greatly
improves performance for large page tables (which is common for large
buffers in a heavily fragmented system).
In my testing, this optimization gave a ~10% speedup when unmapping 64K.
Change-Id: Iaa2b211e730dad6bd9235ef98dd2a89cf541e663
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The ARM LPAE io-pgtable provider now has support for .map_sg. Wire it
up.
Change-Id: I2eaea6f245e1059582f3bf04829e9e3f24675782
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
io-pgtable-arm has just gotten support for .map_sg. Add a test to the
suite of self-tests for this.
Change-Id: Iba56bb801c1f9ef151827598022411c95d389faa
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Mapping an entire scatterlist at once is faster than calling iommu_map
on each link individually. Implement .map_sg in the ARM LPAE page table
allocator so that drivers using the allocator can leverage this
performance boost.
Change-Id: I77f62a2566058693c3f58fc0b05d715a780ae5d8
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The default implementation of map_sg (provided by default_iommu_map_sg
from the IOMMU API layer) actually just calls iommu_map repeatedly.
Often times there are locks or other initial setup in the implementation
of iommu_map, so there's opportunity to drastically improve performance
by factoring that out of the loop. For this reason, io-pgtable
providers might want to implement .map_sg. Add the declaration to the
io_pgtable_ops structure.
Change-Id: I5aa5e4e24a68e303ce2c005dc1dd0b33d5635ab3
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful in IOMMU drivers. Export it.
Change-Id: I4c423d256312250f1e33ca8d64dfe1626f008b5e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Previously we were relying on the callers of the tlb invalidation
functions to enable clocks. However, the new io-pgtable framework
doesn't know anything about clocks and calls directly into the tlb
invalidation functions, so we need to put the clocks enable/disable
inside the tlb functions themselves.
Change-Id: I73a55e3f78deb3501df5615e22cd298fba6e4551
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The newly-introduced smmu_domain.init_mutex can be used everywhere that
we're currently using smmu_domain.lock to protect smmu_domain.smmu.
Move to `init_mutex' and delete `lock'.
Change-Id: I92aa237e72cafd0c8fa39fe6279c44bd194c11b4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some clients need to get the base address of the page table for a given
IOMMU domain. This functionality is provided by the
DOMAIN_ATTR_PT_BASE_ADDR domain attribute. Implement it in the ARM SMMU
driver by returning ttbr[0] from the page table config.
Change-Id: Ie9b0241c7c7df18b25761bae79c1be1a283ff3a4
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The pgtbl_cfg object has a few handy properties that we'd like to make
use of later (returning the pgd in a domain attribute, for example).
Keep track of the domain pgtbl_cfg in the domain structure.
Change-Id: Icce9638a270ca98d6ed6d61b3ef1c35d42a869dc
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The quirk causes the Non-Secure bit to be set in all page table entries.
Change-Id: I937fb7dec4214eca33f8014c664cfc5c99cb0027
Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: c896c132b0
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The ARM SMMU can walk LPAE page tables, so make use of the generic
allocator.
Change-Id: I7acd0e6c6e8046a4f0497a8fbdba46986192f83f
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: 518f713624
[mitchelh: merge with our changes to domain locking, non-shareable
page tables, and pte zero'ing, some of which will need to be added
back now in the generic allocator code]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This patch adds a series of basic self-consistency tests to the ARM LPAE
IO page table allocator that exercise corner cases in map/unmap, as well
as testing all valid configurations of pagesize, ias and stage.
Change-Id: I703df977b7e5914e0ccf9aaca2174cf5956dd604
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: fe4b991dcd
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
A number of IOMMUs found in ARM SoCs can walk architecture-compatible
page tables.
This patch adds a generic allocator for Stage-1 and Stage-2 v7/v8
long-descriptor page tables. 4k, 16k and 64k pages are supported, with
up to 4-levels of walk to cover a 48-bit address space.
Change-Id: I32740cfa795c55e0d3683b42105b4f49c9dcf984
Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: e1d3c0fd70
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This patch introduces a generic framework for allocating page tables for
an IOMMU. There are a number of reasons we want to do this:
- It avoids duplication of complex table management code in IOMMU
drivers that use the same page table format
- It removes any coupling with the CPU table format (and even the
architecture!)
- It defines an API for IOMMU TLB maintenance
Change-Id: Ia7c22397f0afe94898cd91d8d9c30be985993d1d
Tested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: fdb1d7be7c
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit c8690d83e1e1b09d55ed75d6776efe42915b9c95 since it
won't compile when arm-smmu.c moves to the generic page table
allocator (since cfg.pgd is being deleted there). We'll bring this back
in with the appropriate fix once the generic page table allocator patch
is in.
Change-Id: I956d902999a05653e70f7cb7a90ac5010f9926de
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
When domains are set with the DOMAIN_ATTR_NESTING flag, we must ensure
that we allocate them to stage-2 context banks if the hardware permits
it.
This patch adds support for the attribute to the ARM SMMU driver, with
the actual stage being determined depending on the features supported
by the hardware.
Change-Id: I17e027388efbeaae65886160f35a5f8068fd7734
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: c752ce45b2
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[mitchelh: resolved conflicts with our existing domain attr
setter/getter, and locking differences]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently the driver registers IOMMU bus operations for all busses even
if no ARM SMMU is present on a system. Depending on the driver probing
order this prevents the driver for the real IOMMU to register itself as
the bus-wide IOMMU.
Change-Id: I345b0e922ae3d0d71622217fd869747d25ae2bb4
Signed-off-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: 0e7d37adbe
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Exposing the XN flag of the SMMU driver as IOMMU_NOEXEC instead of
IOMMU_EXEC makes it enforceable, since for IOMMUs that don't support
the XN flag pages will always be executable.
Change-Id: Ic6b1f0ae8873035dcb798e9c996396fb2bc4921e
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: a720b41c41
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[mitchelh: context fixups]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
In ["iommu/arm-smmu: fix architecture version detection"] we added the
ability to determine the SMMU architecture version from DT match data by
using of_match_node, but neglected to check the return value for NULL.
Fix this.
Change-Id: Ib9049f2b2f1204b52bc07ba8caf075449af4cd35
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit e6e701ddb09b2b6b43ffc7ff788701eda03703b8. This code
is about to be ripped out in favor of the generic io-pgtable code, so
leaving it in will only cause merged conflicts.
Don't revert the introduction of the DOMAIN_ATTR_SECURE_VMID attribute
to avoid unnecessary churn in client code.
Change-Id: I58af07686f11b14a82d44bd7999ba50127530f1f
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit fd3161483bd7af420b503a8e63d3c1f24a7cf936. This code
is about to be ripped out in favor of the generic io-pgtable code, so
leaving it in will only cause merged conflicts.
Change-Id: I1404f79f08340d3de4a52155c3c82fae6df6515c
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit 0385f613fc41c394873d64785c073dffdfaf0fc4. This code
is about to be ripped out in favor of the generic io-pgtable code, so
leaving it in will only cause merged conflicts.
Change-Id: Id8474f2a400736c288d9db5f9608eddbd24b0793
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit 433a4b6a37c8f96295a7e80cb603c2318872c1f2. An
equivalent patch is being brought in with the upcoming page table
refactoring, so rip this guy out in preparation.
This also reverts commit af3e3c36d08077856640b28d68c9f99d2188a000, which
introduced users of the IOMMU_NOEXEC flag.
Change-Id: Ibb36f60f2148170d8a01632043bf36b5f8c60777
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Following commit implements mapping of sg virtually contiguously.
"arm64: dma-mapping: map sg lists into the SMMU as virtually
contiguous"
a03f74ef16cc73531795176d3ea8b82b66ed0146
This has been left for ARM (32-bit). Implement the same for ARM.
Change-Id: Ibf67f29a60b8d19e526c4719590f2f473ea9dca5
Signed-off-by: Chintan Pandya <cpandya@codeaurora.org>
__hvc should take 11 arguments instead of 10. Add the missing x7
argument to avoid compilation error if the driver gets enabled on
32bit targets.
Change-Id: I09985235fdbfb64e81743a71ceb2764c32d3a3b1
Signed-off-by: Pratik Patel <pratikp@codeaurora.org>
Add API support for calling into the hypervisor. This will allow
various drivers to avail hypervisor services.
Change-Id: I0a0e8f8fe13a550ad20c5421b712e207933c82f3
Signed-off-by: Pratik Patel <pratikp@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
