Disable the SMMU stalls on faulty transactions by terminating it using
the CB.SCTLR register. Faulty transaction on a CB causes the back
pressure on the QSMMU transaction pipe which inturn affect the other
masters using the same SMMU. Pass the DOMAIN_ATTR_CB_STALL_DISABLE flag
to iommu_domain_set_attr api in order to use this feature.
Change-Id: Ib8fa35854ff24e0295d330c5fb79375f58c3e841
Signed-off-by: Charan Teja Reddy <charante@codeaurora.org>
Allow a domain to opt into allocating and maintaining a TTBR1
pagetable. The size of the TTBR1 region will be the same as
the TTBR0 size with the sign extension bit set on the highest
bit in the region.
By example, given a TTBR0/TTBR1 virtual address range of 36
bits the memory map will look like this:
TTBR0 [0x000000000:0x7FFFFFFFF]
TTBR1 [0x800000000:0xFFFFFFFFF]
The map/unmap operations will automatically use the appropriate
pagetable for the given iova.
Change-Id: Ic0dedbad2b2c58cd9c47ce31356472e0463d4228
Signed-off-by: Jordan Crouse <jcrouse@codeaurora.org>
Add a domain attribute to allow a client to force
the page table of a context bank to be coherent.
Change-Id: I5c179e0a5b76ab1ab47ecc271f463df77470db0f
Signed-off-by: Liam Mark <lmark@codeaurora.org>
Add a read-only domain attribute for querying whether or not the
page table of an IOMMU is coherent.
Change-Id: Ifa70b155348ef3d111a41fcad47eccb00d68d427
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Support querying whether an IOVA is mapped as coherent.
Change-Id: Icd122348e309495b09bcf77628aa1746f0255305
Signed-off-by: Liam Mark <lmark@codeaurora.org>
For performance reasons there are clients who would like to move
from stage 1 bypass to stage 1 enabled without having to stop their
device.
Currently clients need to stop their device because they have to
create the required stage 1 mappings before re-enabling the device.
Add the new DOMAIN_ATTR_EARLY_MAP domain attribute to allow clients
to create stage 1 mappings after attaching but before enabling
stage 1 translations.
If the clients set the DOMAIN_ATTR_EARLY_MAP domain attribute to 1
before attaching then then once they attach the SMMU driver won't
enable stage 1 translations. This gives the client the opportunity to
create the required early mappings (for example using iommu_map).
When the client has finished creating the necessary early mappings
the client can then set the DOMAIN_ATTR_EARLY_MAP domain attribute
to 0, this will in turn enable stage 1 translations.
Change-Id: I9e95c5d2130f1d371e201eac69dec140cc773b1f
Signed-off-by: Liam Mark <lmark@codeaurora.org>
The DOMAIN_ATTR_COHERENT_HTW_DISABLE IOMMU domain attribute is no longer
used. Rip it out.
Change-Id: Icc8b81e3020276a1a2c0ed7798eb36568cb866db
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Add an "inline" so the compiler no longer gives a defined-but-not-used
warning.
Change-Id: I226242bf994e659d8018a885f20c3a3f80b9b8e5
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
For certain DMA API implementations, the overhead of going through the
IOMMU framework is too much. Such an implementation might want to
perform some rudimentary page table management using bits of information
from the underlying page tables. Add a domain attribute and structure
for querying this type of information. For now, the only information
supported is the kernel virtual address of the PMDs (assumed to be
virtually contiguous).
CRs-Fixed: 997751
Change-Id: I29d31e9649c24d30a5a7ffaa4b238a0203846594
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
There are certain use cases where it might be necessary to leave the
IOMMU's configuration clocks on. This might happen in places where an
IOMMU's clocks might not be known. A good example of this would be a
test library that needs to be able to do TLB invalidation from atomic
context. It would need to enable clocks up front (outside of atomic
context) and leave them on for the duration of the test.
Add some ops for enabling and disabling configuration clocks.
CRs-Fixed: 997751
Change-Id: I95056952f60494fe5745f2183f9af8aab3a40315
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some higher-level DMA mappers might be able to squeeze out more
performance if TLB invalidation can be delegated to them, since they
might have more knowledge about when a stale TLB is problem than the
IOMMU driver. Add a callback for this purpose that can be implemented
by individual IOMMU drivers.
CRs-Fixed: 997751
Change-Id: If817f5514fdd5d24b9c592440760b81b88ec71a8
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some IOMMU drivers offer "fast" page table management routines for
special cases. There is often a trade-off with memory, etc. with these
so make their usage explicit with a domain attribute.
CRs-Fixed: 997751
Change-Id: Ia9f8ad6d924b294b6758970da2e9767f183b5649
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we use a single pgsize_bitmap per IOMMU driver. However, some
IOMMU drivers might service different IOMMUs with different supported
page sizes. Some drivers might also want to restrict page sizes for
different use cases. Support these use cases by adding a
.get_pgsize_bitmap function to the iommu_ops which can optionally be
used by the driver to return a domain-specific pgsize_bitmap.
CRs-Fixed: 997751
Change-Id: I46d70733be647599e148fe52258a4d8f009ac48a
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Certain configurations may require nested translation, but may
want to avoid software performance limitation on map/unmap
operations.
Change-Id: I69bed6cc9d801709849ebc91b76d2a9d6649865a
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
To allow msm_iommu kernel test compilation, add a
iommu_unmap_range declaration and the corresponding stub
function.
Change-Id: If80918cc01afa7d0df2673f4e1962989af26f4bf
Signed-off-by: Pratik Patel <pratikp@codeaurora.org>
Rename that function to pci_device_group() and export it, so
that IOMMU drivers can use it as their device_group
call-back.
Change-Id: Ic54268d9854dd2eeba53ca9f9635d0287bfc7f0f
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
That call-back is currently unused, change it into a
call-back function for finding the right IOMMU group for a
device.
This is a first step to remove the hard-coded PCI dependency
in the iommu-group code.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Taking inspiration from the existing arch/arm code, break out some
generic functions to interface the DMA-API to the IOMMU-API. This will
do the bulk of the heavy lifting for IOMMU-backed dma-mapping.
Since associating an IOVA allocator with an IOMMU domain is a fairly
common need, rather than introduce yet another private structure just to
do this for ourselves, extend the top-level struct iommu_domain with the
notion. A simple opaque cookie allows reuse by other IOMMU API users
with their various different incompatible allocator types.
Change-Id: I4a49976c4e496025b2a2b2b9ef749666a239294b
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Add changes only in iommu.h]
Using __printf attributes helps to detect several format string issues
at compile time (even though -Wformat-security is currently disabled in
Makefile). For example it can detect when formatting a pointer as a
number, like the issue fixed in commit a3fa71c40f ("wl18xx: show
rx_frames_per_rates as an array as it really is"), or when the arguments
do not match the format string, c.f. for example commit 5ce1aca814
("reiserfs: fix __RASSERT format string").
To prevent similar bugs in the future, add a __printf attribute to every
function prototype which needs one in include/linux/ and lib/. These
functions were mostly found by using gcc's -Wsuggest-attribute=format
flag.
Change-Id: I17371bb8a1cc40cc0b56fcdded609cc24fe7e261
Signed-off-by: Nicolas Iooss <nicolas.iooss_linux@m4x.org>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Felipe Balbi <balbi@ti.com>
Cc: Joel Becker <jlbec@evilplan.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
[pdaly@codeaurora.org Resolve minor conflicts]
This function can be called by an IOMMU driver to request
that a device's default domain is direct mapped.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Add two new functions to the IOMMU-API to allow the IOMMU
drivers to export the requirements for direct mapped regions
per device.
This is useful for exporting the information in Intel VT-d's
RMRR entries or AMD-Vi's unity mappings.
Change-Id: Iab55341a8526084a5110dc5a2d4448fd46e3296a
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
This function can be used to request the current domain a
device is attached to.
Change-Id: I2c88ad3b81d8c9ab5526843ff39e7ff35c74a0fb
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
All drivers have been converted to the new domain_alloc and
domain_free iommu-ops. So remove the old ones and get rid of
iommu_domain->priv too, as this is no longer needed when the
struct iommu_domain is embedded in the private structures of
the iommu drivers.
Tested-by: Thierry Reding <treding@nvidia.com>
Tested-by: Heiko Stuebner <heiko@sntech.de>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
This allows to handle domains differently based on their
type in the future. An IOMMU driver can implement certain
optimizations for DMA-API domains for example.
The domain types can be extended later and some of the
existing domain attributes can be migrated to become domain
flags.
Tested-by: Thierry Reding <treding@nvidia.com>
Tested-by: Heiko Stuebner <heiko@sntech.de>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
These new call-backs defer the allocation and destruction of
'struct iommu_domain' to the iommu driver. This allows
drivers to embed this struct into their private domain
structures and to get rid of the domain_init and
domain_destroy call-backs when all drivers have been
converted.
Tested-by: Thierry Reding <treding@nvidia.com>
Tested-by: Heiko Stuebner <heiko@sntech.de>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
This patch adds a new function to the iommu_ops structure to allow an
OF device to be added to a specific IOMMU instance using the recently
merged generic devicetree binding for IOMMUs. The callback (of_xlate)
takes a struct device representing the master and an of_phandle_args
representing the IOMMU and the correspondong IDs for the new master.
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
IOMMU drivers must be initialised before any of their upstream devices,
otherwise the relevant iommu_ops won't be configured for the bus in
question. To solve this, a number of IOMMU drivers use initcalls to
initialise the driver before anything has a chance to be probed.
Whilst this solves the immediate problem, it leaves the job of probing
the IOMMU completely separate from the iommu_ops to configure the IOMMU,
which are called on a per-bus basis and require the driver to figure out
exactly which instance of the IOMMU is being requested. In particular,
the add_device callback simply passes a struct device to the driver,
which then has to parse firmware tables or probe buses to identify the
relevant IOMMU instance.
This patch takes the first step in addressing this problem by adding an
early initialisation pass for IOMMU drivers, giving them the ability to
store some per-instance data in their iommu_ops structure and store that
in their of_node. This can later be used when parsing OF masters to
identify the IOMMU instance in question.
Change-Id: Ie7051c90c0f2e790d016eb465f10ddda48b99cd1
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
[pdaly@codeaurora.org Resolve minor conflicts]
Some IOMMUs accept an IOMMU_NOEXEC protection flag in addition to
IOMMU_READ and IOMMU_WRITE. Expose this as an IOMMU capability.
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
During development, any IOMMU faults most likely need to be debugged
immediately. Allowing the system to continue to run has a tendency to
mask the original problem, which makes debugging more difficult. Some
drivers might want to make all faults fatal by default, unless
explicitly requested otherwise by a client. Add a domain attribute for
this.
Change-Id: I454b143ea3cae1f3de36e0888afd15936aa63ce1
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It might be useful for IOMMU clients to peek and poke at their IOMMU's
registers, but knowing how to access those registers is really the job
of the IOMMU driver (it might need to enable specific clocks and
regulators, for example). Provide an API to read and write IOMMU
registers that can be implemented by the driver.
Change-Id: I5b2f19225f8bd258278780ff24b4ea96460857aa
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
IOMMU drivers might want more control over the types of faults being
triggered with iommu_trigger_fault. Add a flags parameter that can be
used to provide more control over the types of faults that can be
triggered.
Change-Id: I2f21b383437430e957ab52070d3575e8cb3dee90
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some IOMMUs (like the ARM SMMU) can raise "external faults" in addition
to translation and permission faults. Add an IOMMU fault flag so that
this fault type can be propagated to client fault handlers.
Change-Id: Ia476ed6c727f5f01011747f30b3a74ebf0fc07f7
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently we're creating an "iommu" debugfs directory from the
iommu-debug code. Other IOMMU modules might want to make use of this
same directory, so create it from the IOMMU framework code itself.
Change-Id: I679fdfc34ba5fcbd927dc5981438c6fabcfa3639
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some devices can support per-process iommu domains, which are used
asynchronously by the hardware which directly updates to the TTBR0
and CONTEXTIDR registers.
Add DOMAIN_ATTR_DYNAMIC to do indicate a domain may be used dynamically.
This attribute must be set before attaching, as it changes what
domain and hardware configuration is done by the iommu driver.
Before attaching any dynamic domains, the driver must first attach
a non-dynamic domain to do initial hardware configuration.
A simplified example:
void use_domain(struct iommu_domain *domain, struct job *job)
{
u64 ttbr0;
u32 contextidr;
iommu_domain_get_attr(domain, DOMAIN_ATTR_TTBR0, &ttbr0);
iommu_domain_get_attr(domain, DOMAIN_ATTR_CONTEXTIDR, &contextidr);
/*
* Schedule work on the hardware, which is time sliced between
* clients. Each client has a separate iommu domain and when
* a job is run, the hardware first programs TTBR0 and
* CONTEXTIDR to use the appropriate domain.
*/
submit_job(ttbr0, contextidr, job);
}
void init(void)
{
struct iommu_domain *master_domain, *dyn_domain1, *dyn_domain2;
int dynamic = 1;
master_domain = iommu_domain_alloc(bus);
dyn_domain1 = iommu_domain_alloc(bus);
dyn_domain2 = iommu_domain_alloc(bus);
iommu_attach_device(base_domain, dev);
iommu_domain_set_attr(dyn_domain1, DOMAIN_ATTR_DYNAMIC, &dynamic);
iommu_domain_set_attr(dyn_domain2, DOMAIN_ATTR_DYNAMIC, &dynamic);
iommu_attach_device(dyn_domain1, dev);
iommu_attach_device(dyn_domain2, dev);
while (keep_going) {
iommu_map(dyn_domain1, ...);
iommu_map(dyn_domain2, ...);
use_domain(dyn_domain1, job1);
use_domain(dyn_domain2, job2);
iommu_unmap(dyn_domain1, ...);
iommu_unmap(dyn_domain2, ...);
}
iommu_detach_device(dyn_domain2, dev);
iommu_detach_device(dyn_domain1, dev);
iommu_detach_device(master_domain, dev);
}
Change-Id: Ic4fa0a831751eb4b10fff5d9aec28a411856fbd1
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
In the ARM SMMU architecture, pagetable programming is controlled
by the TTBR0 and CONTEXTIDR registers. The layout of these
registers varies depending on the pagetable format in use.
In particular, the ASID (address space ID) field is found in
CONTEXTIDR when using V7S format and in the top bits of TTBR0
for V7L and V8L.
Some drivers need to program hardware to switch domains on the
fly. These attributes allow the correct settings to be determined
by querying the domain rather than directly reading registers and
making assumptions about the pagetable format. The domain must be
attached before TTBR0 and CONTEXTIDR may be queried.
The PROCID attribute allows driver set a debug field in the
CONTEXTIDR register. This attribute may only be set before
attaching, but may be queried at any time. The SMMU hardware
doesn't use the contents of this field, but debug can be simpler
if each domain stores a unique value in it.
Change-Id: I175aa78fee02c3e4e0071496d9cc2b8841ff9e3c
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
It can be useful to trigger an IOMMU fault during development and
debugging. Add support to the IOMMU framework to do so.
Change-Id: I908c9f5b52c6abe937f031de546d290027ba64b5
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Some IOMMU hardware implementations provide hardware translation
operations that can be useful during debugging and development. Add a
function for this purpose along with an associated op in the iommu_ops
structure so that drivers can implement it.
Change-Id: I54ad5df526cdce05f8e04206a4f01253b3976b48
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
After attaching a domain, this attribute may be
queried to determine which hardware context bank
was assigned.
Change-Id: I31e674672041103007fcaff3f83a0cc2c33a4a6d
Signed-off-by: Jeremy Gebben <jgebben@codeaurora.org>
On some platforms, certain IOMMU hardware state
might be cleared before reaching client fault handlers,
making it difficult for client fault handlers to do much
useful processing. Add some flags so that this information
can be passed to client fault handlers directly, rather
than expecting clients to read the hardware themselves.
Also provide a mechanism for client fault handlers to
request that the IOMMU driver not clear the fault or
resume/retry the faulting transaction. The client fault
handler can return -EBUSY to request this behavior.
Change-Id: I9780beb52b4257fff99d708a493173c9fe0a9d8a
Signed-off-by: Sushmita Susheelendra <ssusheel@codeaurora.org>
Add ftrace start and end logging for map, iommu_map_sg and unmap
in order to facilitate performance testing.
Change-Id: I9ddf241ffa6cf519f6abece7b0820640f5ce1975
Signed-off-by: Liam Mark <lmark@codeaurora.org>
The .dma_supported() checks to see if the device can support DMA to the
memory described by the mask. Add support for this operation in the
iommu layer
Change-Id: Icf37b9540aa68c2be3fd603a48402d6fcccd8208
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Certain clients need map and unmap operations to be in the atomic
context. This currently doesn't work since we need to prepare clocks
and enable regulators (for tlb invalidation), and take a mutex (to
protect the smmu_domain->smmu instance), which are all sleeping
operations.
Fix this by introducing a domain attribute to indicate that map and
unmap should be atomic. When this domain attribute is set we assume:
(1) clocks are already prepared
(2) regulators are already on
(3) the domain is attached to an SMMU
(1) and (2) allow us to skip preparing clocks and enabling regulators
during map and unmap and are taken care of internally in the
driver. (3) is a contract with clients that allows us to skip taking
the mutex that protects smmu_domain->smmu since we assume that the SMMU
instance is valid and will not be going away. If clients break this
contract then there are race conditions all over the place that will
eventually result in a NULL dereference (specifically, of
smmu_domain->smmu).
Change-Id: I3e21df02f7b0cd116fb558715eea16f2119f1d03
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
It can be useful in IOMMU drivers. Export it.
Change-Id: I4c423d256312250f1e33ca8d64dfe1626f008b5e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Exposing the XN flag of the SMMU driver as IOMMU_NOEXEC instead of
IOMMU_EXEC makes it enforceable, since for IOMMUs that don't support
the XN flag pages will always be executable.
Change-Id: Ic6b1f0ae8873035dcb798e9c996396fb2bc4921e
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-commit: a720b41c41
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[mitchelh: context fixups]
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
This reverts commit 433a4b6a37c8f96295a7e80cb603c2318872c1f2. An
equivalent patch is being brought in with the upcoming page table
refactoring, so rip this guy out in preparation.
This also reverts commit af3e3c36d08077856640b28d68c9f99d2188a000, which
introduced users of the IOMMU_NOEXEC flag.
Change-Id: Ibb36f60f2148170d8a01632043bf36b5f8c60777
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
[pdaly@codeaurora.org Resolve minor conflicts]
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Exposing the XN flag of the SMMU driver as IOMMU_NOEXEC instead of
IOMMU_EXEC makes it enforceable, since for IOMMUs that don't support
the XN flag pages will always be executable.
Change-Id: Ib785acd8a188fa95aea9991116139a392862764e
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Git-repo: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
Git-commit: a720b41c41
[rvaswani@codeaurora.org: resolve trivial merge conflicts]
Signed-off-by: Rohit Vaswani <rvaswani@codeaurora.org>
This is used for indicating device memory type for a DMA transaction. IOMMU
driver would set up attributes indicationg access to device memory.
CRs-Fixed: 792402
Change-Id: Ifd6f2288353ef737ff4c15d9bd1514c66af8fdd2
Signed-off-by: Varun Sethi <Varun.Sethi@freescale.com>
[neetid: changed IOMMU_DEVICE to be on bit 5 instead of bit 4]
Patch-mainline: iommu @ 10/06/14, 10:28
Signed-off-by: Neeti Desai <neetid@codeaurora.org>
