Use the information exported by the IOMMU drivers to create
direct mapped regions in the default domains.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Add two new functions to the IOMMU-API to allow the IOMMU
drivers to export the requirements for direct mapped regions
per device.
This is useful for exporting the information in Intel VT-d's
RMRR entries or AMD-Vi's unity mappings.
Change-Id: Iab55341a8526084a5110dc5a2d4448fd46e3296a
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
This function can be used to request the current domain a
device is attached to.
Change-Id: I2c88ad3b81d8c9ab5526843ff39e7ff35c74a0fb
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
Make use of the default domain and re-attach a device to it
when it is detached from another domain. Also enforce that a
device has to be in the default domain before it can be
attached to a different domain.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
This patch changes the behavior of the iommu_attach_device
and iommu_detach_device functions. With this change these
functions only work on devices that have their own group.
For all other devices the iommu_group_attach/detach
functions must be used.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
The default domain will be used (if supported by the iommu
driver) when the devices in the iommu group are not attached
to any other domain.
Change-Id: I9bb6a8619c66694e5fef37a46a53cce8c99d44ec
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
Make sure we call the ->remove_device call-back on all
devices already initialized with ->add_device when the bus
initialization fails.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Write a message to the kernel log when a device is added or
removed from a group and add debug messages to group
allocation and release routines.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Including the function name is only useful for debugging
messages. They don't belong into other messages from the
iommu core.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Changing force_stage dynamically isn't supported by the driver and it
also doesn't make a whole lot of sense to change it once the SMMU is up
and running.
This patch makes the sysfs entry for the parameter read-only.
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
The ATS1* address translation registers only support being written
atomically - in SMMUv2 where they are 64 bits wide, 32-bit writes to
the lower half are automatically zero-extended, whilst 32-bit writes
to the upper half are ignored. Thus, the current logic of performing
64-bit writes as two 32-bit accesses is wrong.
Since we already limit IOVAs to 32 bits on 32-bit ARM, the lack of a
suitable writeq() implementation there is not an issue, and we only
need a little preprocessor ugliness to safely hide the 64-bit case.
Change-Id: Ice82b1276d30605d335f9400f8cc3da3e3348bb6
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
iommu_group_alloc might be called very early in case of iommu controllers
activated from of_iommu, so ensure that this part of subsystem is ready
when devices are being populated from device-tree (core_initcall seems to
be okay for this case).
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Javier Martinez Canillas <javier.martinez@collabora.co.uk>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Stage 1 translation is controlled by two sets of page tables (TTBR0 and
TTBR1) which grow up and down from zero respectively in the ARMv8
translation regime. For the SMMU, we only care about TTBR0 and, in the
case of a 48-bit virtual space, we expect to map virtual addresses 0x0
through to 0xffff_ffff_ffff.
Given that some masters may be incapable of emitting virtual addresses
targetting TTBR1 (e.g. because they sit on a 48-bit bus), the SMMU
architecture allows bit 47 to be sign-extended, halving the virtual
range of TTBR0 but allowing TTBR1 to be used. This is controlled by the
SEP field in TTBCR2.
The SMMU driver incorrectly enables this sign-extension feature, which
causes problems when userspace addresses are programmed into a master
device with the SMMU expecting to map the incoming transactions via
TTBR0; if the top bit of address is set, we will instead get a
translation fault since TTBR1 walks are disabled in the TTBCR.
This patch fixes the issue by disabling sign-extension of a fixed
virtual address bit and instead basing the behaviour on the upstream bus
size: the incoming address is zero extended unless the upstream bus is
only 49 bits wide, in which case bit 48 is used as the sign bit and is
replicated to the upper bits.
Change-Id: Iaa142beaeccd57b3ba1718ae7ea6657fe6e5d8c9
Cc: <stable@vger.kernel.org> # v4.0+
Reported-by: Varun Sethi <varun.sethi@freescale.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
All drivers have been converted to the new domain_alloc and
domain_free iommu-ops. So remove the old ones and get rid of
iommu_domain->priv too, as this is no longer needed when the
struct iommu_domain is embedded in the private structures of
the iommu drivers.
Tested-by: Thierry Reding <treding@nvidia.com>
Tested-by: Heiko Stuebner <heiko@sntech.de>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Implement domain_alloc and domain_free iommu-ops as a
replacement for domain_init/domain_destroy.
Change-Id: Ie503be088fe85a1a8e8242d133d3f7c0d61cb458
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Use to_iommu_domain()]
Check for the new __IOMMU_DOMAIN_PAGING flag before calling
into the iommu drivers ->map and ->unmap call-backs.
Change-Id: If4c6a7133fd33797f7e685629ef5793de3288e32
Tested-by: Thierry Reding <treding@nvidia.com>
Tested-by: Heiko Stuebner <heiko@sntech.de>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
This allows to handle domains differently based on their
type in the future. An IOMMU driver can implement certain
optimizations for DMA-API domains for example.
The domain types can be extended later and some of the
existing domain attributes can be migrated to become domain
flags.
Tested-by: Thierry Reding <treding@nvidia.com>
Tested-by: Heiko Stuebner <heiko@sntech.de>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
These new call-backs defer the allocation and destruction of
'struct iommu_domain' to the iommu driver. This allows
drivers to embed this struct into their private domain
structures and to get rid of the domain_init and
domain_destroy call-backs when all drivers have been
converted.
Tested-by: Thierry Reding <treding@nvidia.com>
Tested-by: Heiko Stuebner <heiko@sntech.de>
Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Although we set TCR.T1SZ to 0, the input address range covered by TTBR1
is actually calculated using T0SZ in this case on the ARM SMMU. This
could theoretically lead to speculative table walks through physical
address zero, leading to all sorts of fun and games if we have MMIO
regions down there.
This patch avoids the issue by setting EPD1 to disable walks through
the unused TTBR1 register.
Change-Id: I766a0e19714b7f4e6659331ae0772efc28b95224
Signed-off-by: Will Deacon <will.deacon@arm.com>
[pdaly@codeaurora.org Use upstream version]
IOMMU groups for PCI devices can correspond to multiple DMA aliases due
to things like ACS and PCI quirks.
This patch extends the ARM SMMU ->add_device callback so that we
consider all of the DMA aliases for a PCI IOMMU group, rather than
creating a separate group for each Requester ID.
Change-Id: I3b8e81e17447cc0bbd0f9d299b5494c390372002
Signed-off-by: Will Deacon <will.deacon@arm.com>
Since we use dma_map_page() as an architecture-independent means of
making page table updates visible to non-coherent SMMUs, we need to
have a suitable DMA mask set to discourage the DMA mapping layer from
creating bounce buffers and flushing those instead, if said page tables
happen to lie outside the default 32-bit mask.
Change-Id: Ia97122cef853cb48c9ad45a2a35717dd85cd9764
Tested-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[will: added error checking]
Signed-off-by: Will Deacon <will.deacon@arm.com>
[pdaly@codeaurora.org Take upstream version]
The VMID16 (8.1) extension to SMMUv2 added a 16-bit VMID16 field to the
CBA2R registers. Unfortunately, if software writes this field as zero
after setting an 8-bit VMID in a stage-2 CBAR, then the VMID may also be
overwritten with zero on some early implementations (the architecture
was later updated to fix this issue).
This patch ensures that we initialise CBA2R before CBAR, therefore
ensuring that the VMID is set correctly.
Tested-by: Manish Jaggi <mjaggi@caviumnetworks.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
This patch is a fix to "iommu/arm-smmu: add support for iova_to_phys
through ATS1PR".
According to ARM documentation, translation registers are optional even
in SMMUv1, so ID0_S1TS needs to be checked to verify their presence.
Also, we check that the domain is a stage-1 domain.
Change-Id: I2164ddb3806f941d21463731d0991ce1a83a5221
Signed-off-by: Baptiste Reynal <b.reynal@virtualopensystems.com>
Acked-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Various build/boot bots have reported WARNs being triggered by the ARM
iopgtable LPAE self-tests on i386 machines.
This boils down to two instances of right-shifting a 32-bit unsigned
long (i.e. an iova) by more than the size of the type. On 32-bit ARM,
this happens to give us zero, hence my testing didn't catch this
earlier.
This patch fixes the issue by using DIV_ROUND_UP and explicit case to
to avoid the erroneous shifts.
Reported-by: Fengguang Wu <fengguang.wu@intel.com>
Reported-by: Huang Ying <ying.huang@intel.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
iommu_map() calls trace_map() with iova and size. trace_map()
should report original iova and original size as opposed to
iova and size after they get changed during mapping. size is
always zero at the end of mapping which is useless to report
and iova as it gets incremented, it is not as useful as the
original iova. Change iommu_map() to call trace_map() to
report original iova and original size.
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Reported-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
Currently map and unmap are implemented as events under a
common trace class declaration. The common class forces
trace_unmap() to require a bogus physical address argument
that it doesn't use. Changing unmap to report unmapped size
will provide useful information for debugging. Remove common
map_unmap trace class and change map and unmap into separate
events as opposed to events under the same class to allow for
differences in the reporting information. In addition, map and
unmap are changed to handle size value as size_t instead of int
to match the passed size value and avoid overflow.
Change-Id: Ic231e28c2784fbfe41a6749175cc390920165e81
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Suggested-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Keep map_unmap class]
iommu_unmap() calls trace_unmap() with changed iova and original
size. trace_unmap() should report original iova instead. Change
iommu_unmap() to call trace_unmap() with original iova.
Change-Id: Iedbaf6ddef566626f5c3dd656a8c1b8cf8c15663
Signed-off-by: Shuah Khan <shuahkh@osg.samsung.com>
Reported-by: Alex Williamson <alex.williamson@redhat.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
[pdaly@codeaurora.org Resolve minor conflicts]
If the IOMMU supports pages smaller than the CPU page size, segments
which lie at offsets within the CPU page may be mapped based on the
finer-grained IOMMU page boundaries. This minimises the amount of
non-buffer memory between the CPU page boundary and the start of the
segment which must be mapped and therefore exposed to the device, and
brings the default iommu_map_sg implementation in line with
iommu_map/unmap with respect to alignment.
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
IOMMU drivers can be initialized from of_iommu helpers. Such drivers don't
need to provide device_add callbacks to operate properly, so there is no
need to fail initialization if the callback is missing.
Acked-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Marek Szyprowski <m.szyprowski@samsung.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
This patch adds a new function to the iommu_ops structure to allow an
OF device to be added to a specific IOMMU instance using the recently
merged generic devicetree binding for IOMMUs. The callback (of_xlate)
takes a struct device representing the master and an of_phandle_args
representing the IOMMU and the correspondong IDs for the new master.
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
IOMMU drivers must be initialised before any of their upstream devices,
otherwise the relevant iommu_ops won't be configured for the bus in
question. To solve this, a number of IOMMU drivers use initcalls to
initialise the driver before anything has a chance to be probed.
Whilst this solves the immediate problem, it leaves the job of probing
the IOMMU completely separate from the iommu_ops to configure the IOMMU,
which are called on a per-bus basis and require the driver to figure out
exactly which instance of the IOMMU is being requested. In particular,
the add_device callback simply passes a struct device to the driver,
which then has to parse firmware tables or probe buses to identify the
relevant IOMMU instance.
This patch takes the first step in addressing this problem by adding an
early initialisation pass for IOMMU drivers, giving them the ability to
store some per-instance data in their iommu_ops structure and store that
in their of_node. This can later be used when parsing OF masters to
identify the IOMMU instance in question.
Change-Id: Ie7051c90c0f2e790d016eb465f10ddda48b99cd1
Acked-by: Arnd Bergmann <arnd@arndb.de>
Acked-by: Joerg Roedel <jroedel@suse.de>
Acked-by: Marek Szyprowski <m.szyprowski@samsung.com>
Tested-by: Robin Murphy <robin.murphy@arm.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
[pdaly@codeaurora.org Resolve minor conflicts]
The ARM SMMU supports the IOMMU_NOEXEC protection flag. Add the
corresponding IOMMU capability.
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Some IOMMUs accept an IOMMU_NOEXEC protection flag in addition to
IOMMU_READ and IOMMU_WRITE. Expose this as an IOMMU capability.
Signed-off-by: Antonios Motakis <a.motakis@virtualopensystems.com>
Acked-by: Joerg Roedel <jroedel@suse.de>
Signed-off-by: Will Deacon <will.deacon@arm.com>
When some part of bus_set_iommu fails it should undo any made changes
and not simply leave everything as is.
This includes unregistering the bus notifier in iommu_bus_init when
add_iommu_group fails and also setting the bus->iommu_ops back to NULL.
Signed-off-by: Heiko Stuebner <heiko@sntech.de>
Signed-off-by: Joerg Roedel <jroedel@suse.de>
The IOMMU-API works on page boundarys, unlike the DMA-API
which can work with sub-page buffers. The sg->offset
field does not make sense on the IOMMU level, so force it to
be 0. Do some error-path consolidation while at it.
Signed-off-by: Joerg Roedel <jroedel@suse.de>
A platform_driver does not need to set an owner, it will be populated by the
driver core.
Change-Id: I4382a4b82bba56cf5cd2c2bfe56cdbafeb647cb8
Signed-off-by: Wolfram Sang <wsa@the-dreams.de>
[pdaly@codeaurora.org Resolve minor conflicts]
Print time with ns precision.
Ex:
size iommu_map iommu_unmap
4K 43.021 us 99.479 us
Change-Id: Ib7f3db2be533c77adf0a20de557615a0e2433dd9
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Add a nr_iters file which specifies the number of runs to average
performance over. Alternatively, setting this to 1 allows the user
to collect raw data.
Change-Id: Ifab7215027b597fb2650ab2f0c5065bebe0b8c5d
Signed-off-by: Patrick Daly <pdaly@codeaurora.org>
Allow compilation when CONFIG_ARM_DMA_USE_IOMMU is not
selected by providing necessary stub functions and fix few
mismatch in function declarations.
Following this, remove the changes introduced by
f3d8d1061fb0b146b3f5 ("msm: ipa: add empty implementation
for iommu functions") as they are no longer needed.
Change-Id: I04e3aa63407064e8d9c9550a5cb0a82ede899f00
Signed-off-by: Shiraz Hashim <shashim@codeaurora.org>
The fault handler currently spits out a ton of information that can make
clients' eyes glaze over, and understandably so. One of the most
important pieces of information we obtain from the fault handler is
whether the faulting address is actually mapped (which we check with a
software table walk), but that information is currently not obvious (you
have to know that "soft iova-to-phys=0" means that the software table
walk failed). Fix this by adding a more obvious error message when we
detect an unmapped access.
Change-Id: Ib95a916baf3e73934642676de4852a550ae0f54b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
We're currently waiting for up to 1 second for the TLB sync command to
finish. Unfortunately, this is long enough to trigger a watchdog bark
since the time between the watchdog pet timer interrupt and the bark
interrupt is only 1 second. Fix this by reducing the timeout to 500
milliseconds, which is still more than enough time for the TLB sync
command to complete.
CRs-Fixed: 929880
Change-Id: I467cd5fd9caa24b4f120a7fe8c8f57c73687006b
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The SMMU halt needs to be accessed before regulator is disabled. So move
notification from REGULATOR_EVENT_DISABLE to REGULATOR_EVENT_PRE_DISABLE.
Regulator framework will notify client on other events too, return
NOTIFY_OK.
Change-Id: Ieed950418d5afc6f5b84b13131ef39964f3168e2
Signed-off-by: Jaydeep Sen <jsen@codeaurora.org>
As per the spec, FRSYNRA contains the SID in the lower 16 bits.
Currently we're masking off all but the lower 8 bits, so we could be
losing bits when we print SIDs. Fix this by using a 16 bit mask.
Change-Id: I4a16c169a52d31b43b2f850a5558eecb64b93902
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Before SMMU is powered down, SMMU needs to be in
idle state prior to power collapse. When 'halt' is
received by SMMU, it ensures that no new requests
enters and all outstanding requests are completed
and generates an acknowledgment for halt request.
Add an option to register a notifier on regulator
so that SMMU can be halted/resumed when regulator
is powered down/up. Some of the targets have built
in provision in hardware to halt SMMU when there
are no on going transactions. We should keep this
option disable for those targets.
Change-Id: Ia2f8a934a9d64daefdacd517eb22f09de5eeb273
Signed-off-by: Susheel Khiani <skhiani@codeaurora.org>
We're currently disabling clocks/power immediately after writing to
FSRRESTORE, but if the fault is not handled before we disable the
clocks/power then the fault could be lost. Add a 1-second sleep after
writing to FSRRESTORE to give the interrupt time to fire before turning
clocks/power back off.
Change-Id: Ieeba49686da5da92e43f03c3c593166592184433
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
Currently, in the arm_lpae_map_sg early return path (when neither READ
nor WRITE permissions were specified), we don't store the "partially
mapped" size in the size output parameter, as is expected. This could
result in someone up the stack using an uninitialized variable. Fix
this by storing 0 in *size when there's nothing to do.
Change-Id: I3b0952aa5bcc2810843b7c513c32c3d665169e40
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
The return value of the .map_sg callback is currently undocumented.
Document it.
Change-Id: I195cbfb78eaf173d916dd03190f89b2fe04e144e
Signed-off-by: Mitchel Humpherys <mitchelh@codeaurora.org>
