This change enables the aggressive clock gating along with hibern8 at
every 10ms. As hibern8 enter was already @10ms, we are now merging it
with clock gating which will also be @10ms after this change. This change
is needed to save power for real life usecases.
Change-Id: I297d0f7d1c379f0b402298dc5dddd92ac33f6f25
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Currently we are suspending clock scaling during clock gating which doesn't
allow us to have clock gating timeout lower than clock scaling polling
window. If clock gating timeout is smaller than the clock scaling polling
window then we will mostly suspend the clock scaling before clock scaling
polling window expires and we might get stuck in same state (scaled down
or scaled up) for quite a long time. And for this reason, we have clock
gating timeout (50ms) greater than clock scaling polling window (40ms).
We would like to have aggressive clock gating timeout even lower than the
clock scaling polling window hence this change is decoupling the clock
scaling suspend/resume from clock gate/ungate. We will not suspend the
clock scaling as part of clock gating instead clock scaling context will
schedule scaling suspend work if there are no more pending transfer
requests.
Change-Id: I7c1e7580c63bbbfbad551d17921774afa3d95ca6
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Currently we call the scsi_block_requests()/scsi_unblock_requests()
whenever we want to block/unblock scsi requests but as there is no
reference counting, nesting of these calls could leave us in undesired
state sometime. Consider following call flow sequence:
1. func1() calls scsi_block_requests() but calls func2() before
calling scsi_unblock_requests()
2. func2() calls scsi_block_requests()
3. func2() calls scsi_unblock_requests()
4. func1() calls scsi_unblock_requests()
As there is no reference counting, we will have scsi requests unblocked
after #3 instead of it to be unblocked only after #4. Though we may not
have failures seen with this, we might run into some failures in future.
Better solution would be to fix this by adding reference counting.
Change-Id: I1eaa39c0716cf41120269cf389ebb322e3006da1
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Enabling & disabling host PHY ref_clk generally may have high latencies
hence don't disable them during aggresive clock gating otherwise it could
affect the UFS performance badly.
Change-Id: I1741c75a6295c7c0b0d77ff578ab977213dcc484
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
It is quite possible that we might need multiple clocks to be enabled when
UFS transfers are active and we generally turn these clocks off during
runtime & system suspend. In addition, we also vote these clocks off if
aggressive clock gating feature is enabled. Idle timeout for aggressive
clock gating feature is generally ~50 milliseconds. But turning these
clocks on/off could have huge latencies hence we might only want to turn
off few essential (and low latency) clocks during aggressive clock gating.
This change adds support to let the vendor specific setup_clocks callback
know whether it is called from aggressive clock gating context or normal
clock gating context. Having this context information should help vendor
specific setup_clocks callback to selectively disable clocks.
Change-Id: I5e1523a57bc45a91faef463baac1cea2a2c8d2d6
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Voting/devoting device ref_clk source could be time consuming hence add
support to allow ufs variant to decide if it wants to control ref_clk
during aggressive clock gating or not. This clock should still be gated
off during runtime/system suspend.
Change-Id: Ibec8cc8b3689ff4e2418039e74dbf311e589157e
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Currently we power collapse UFS controller core only during runtime suspend
but we could also power collapse it during clock gating as power collapse
latencies are only few microseconds. This change adds this aggressive power
collapse to save power.
Change-Id: Id550bcbcd044670cec08a6abb7225cb9386993b5
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Some platforms don't support restoring the secure configuration as
secure configuration can never be lost hence scm_restore_sec_cfg() is
expected to be failed for such platforms. This change changes the
failure message level from error to debug.
Change-Id: Ie4b96971d5c4c394576a78444fe5013749375c28
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
If we had called the pm_qos_add_request() with PM QoS request type set to
PM_QOS_REQ_AFFINE_IRQ then freeing up the irq makes the free_irq() to
print out warning with call stack. We don't really need to free up irq
during suspend, disabling it during suspend and reenabling it during
resume should be good enough and that's how it was done on 3.10 kernel
as well.
Change-Id: I58379a05e1f9d5f18b46394d64bdf6f106afa963
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
UFS device ref_clk frequency is set during the one time provisioning,
UFS driver just queries the reference clock frequency from device and
prints it out for informational purpose. Although this seems redundant
as incorrect reference clock frequency would anyway be detected during
bootup phase itself. This patch removes this redundant query and in turn
removes one message from going to serial console.
Change-Id: Id6a4d6d84b05f4b07b5299fd90e78e793e9689dc
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Add debug-fs entry to show current state of UFS PM QoS.
Change-Id: Ic1ae582b5a657f332be8a6f8d3bbcc0434bdf3d3
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Add sys-fs entries to allow user space control PM QoS
latency parameters and enable/disable PM QoS voting.
The entries are added under:
/sys/bus/platform/devices/624000.ufshc/624000.ufshc:ufs_variant/
pm_qos_enable:
write 0 to disable PM QoS, 1 to enable.
Example: "echo 1 > pm_qos_enable"
pm_qos_latency_us:
write the desired value for each cpu group, separated by a comma.
Example: "echo 10,20 > pm_qos_latency_us"
Change-Id: I9797a1e62c4867ab831b4f18cbb1e0ca9834247b
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Signed-off-by: Krishna Konda <kkonda@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Current UFS PM QoS design and implementation do not seem to give
the desired increase in performance. This change revisits the
PM QoS implementation trying to improve performance by making
the following changes:
* de-couple voting from clock scaling decision so voting
occurs from the first request and unvoting on the completion
of the last request regardless to clock scaling state.
Otherwise, suspending the PM QoS voting during the time it takes
to decide on clock up-scaling, seems to degrade random access
performance.
* vote on a per-cluster basis by inspecting the request object's
cpu field. This follows the soft-irq allocation scheme in the
block layer, so the cpu to which the block layer schedules the
soft-irq will not be put into deep LPM.
We should note that PM QoS voting using cpu mask for specific
cpus is a feature of the qcom specific PM QoS implementation.
Change-Id: I427d202aeb45cd284a3bb128e26e519212614801
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Signed-off-by: Krishna Konda <kkonda@codeaurora.org>
[venkatg@codeaurora.org: resolved merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
The new UFS Controller (version 3.0.0) has a few changes compared to
previous versions, such as a new vendor specific register, and
different debug register offsets.
Change-Id: Idf455d2dcbdcdadcb1e809a8349fc9adbe9733dd
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
Some platforms do not enable the BUS_SCALING feature.
For such cases, we should wrap the routines that use this feature
with a configuration flag.
Change-Id: Iced5b6492a470eb5700b50a50de8260b6bb59753
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
The UFS HCI v2.1 includes a few additional registers. This change
updates the HCI register, the UFS version register content and
the Interrupt Status register.
Change-Id: Icfeef0e6aaed064b456fac1cd5531fff1328ae37
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
When we are resuming the UFS device rails in HPM mode, we are first
powering on the VCC rail while VCCQ and VCCQ2 rails still being in LPM
mode. Some UFS devices may take VCC on event as hint that host wants
UFS device to be resumed and may start drawing more power from the
VCCQ/VCCQ2 rails (while they are still in LPM mode) causing voltage drop
on these rails. This change fixes this issue by bringing VCCQ & VCCQ2 rails
out of LPM before powering on VCC rail.
Change-Id: I950591d2a571257e08d7697e2b33dd5968f7333f
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This patch splits the vops by type and creates wrapper
functions for all vops.
Used in order clarify the runtime code and to allow
further types of vops.
This is the first step towards rearchitecting the
variant implementation of the UFS driver - per requirement
from the community. As this will become a full-blown
rearchitecture - incrementing the driver version.
Change-Id: Ifd0a5412a608590bc6c761caf0b3a87aed37f70b
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
[venkatg@codeaurora.org: resolved merge conflicts,
change ufshcd_pltfrm_init to use ufs_hba_variant
instead of ufs_hba_variant_ops]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Adds setter/getter functions to set/get a pointer
to the variant control structure in/from the generic
hba control structure.
Change-Id: I4c7b3967366f394472490410e3880fdc6e3608b4
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
[venkatg@codeaurora.org: use ufshcd_set_variant instead of
ufshcd_bind_variant]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
When compiling with gcc5, it throws below errors:
drivers/scsi/ufs/ufs-qcom.c: In function 'ufs_qcom_hce_enable_notify':
drivers/scsi/ufs/ufs-qcom.c:352:10: warning: switch condition has boolean
value [-Wswitch-bool] error, forbidden warning: ufs-qcom.c:352
This is because we are using the boolean value in switch condition and it
seems to be forbidden by compiler (by default). Fix this by using the
proper enum type instead of boolean.
Change-Id: Iee6950e5986dc51afbf9440cd4773b4d0f02fcdc
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflict]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
We are currently running UFS link in HS-G3 FAST mode during high load
condition for best possible performance and in HS-G2 FAST mode during
low load condition to save power. As we are anyway scaling down from
HS-G3 to HS-G2, we can also change the mode from FAST to FASTAUTO.
So we looked at the performance numbers with HS-G2 FASTAUTO mode and
they are good enough for most of the low bandwidth usecases. But Samsung
UFS memory devices are exception which has really low sequential read
throughput in FAST AUTO mode hence we will only be enabling FAST AUTO mode
for other UFS device vendors.
Change-Id: Ia79da3e82fb87c0e396534979398617388f40af9
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
DME commands such as Hibern8 enter/exit and gear switch generate 2
completion interrupts, one for confirmation that command is received
by local UniPro and 2nd one is the final confirmation after communication
with remote UniPro. Currently both of these completions are registered
as interrupt events which is not quite necessary and instead we can
just wait for the interrupt of 2nd completion, this should reduce
the number of interrupts and could reduce the unnecessary CPU wakeups
to handle extra interrupts.
Change-Id: I5365b8b5cae952fd4bd7bb93cca0cfe76ea8b619
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
UniPro v1.6 specication requires both host and device TX LCC Enable to be
in same state but current driver still attempts to disable just one
side of LCC. With this change, if Host supports UniPro v1.6 (and all
commercial UFS devices anyway supports the UniPro v1.6), we would keep
both host and device TX LCC disabled after the link startup. Keeping LCC
disabled helps avoid multiple compatibility issues between host and device.
Change-Id: I1697089255b63f70a465c8b42a899ee6ab0acd88
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
DME attributes read/write commands need GenSelectorIndex argument and
according to UFS host controller specification, it should specify the
targeted M-PHY data lane.
This is the valid range of GenSelectorIndex for M-PHY attributes:
0 to (2 * PA_MaxDataLanes - 1)
Example (Note: PA_MaxDataLanes is UniPro protocol constant set to 4):
-----------------------------------------
MPHY Access target | GenSelectorIndex
-----------------------------------------
TX Lane0 | 0
TX Lane1 | 1
RX Lane0 | 4
RX Lane1 | 5
-----------------------------------------
This change makes sure that correct GenSelectorIndex is passed for M-PHY
attributes while it reading via debugfs.
Change-Id: If09e4271958e4bb18d315446cef253caec75584a
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Quite a few UFS functions (especially those which are invoked in debugfs
context) are calling the ufshcd_hold_all() first before runtime resuming
the UFS host controller. As runtime suspend might have already power
collapsed UFS core controller, attempt to enable clocks (as done by
ufshcd_hold_all()) in power collapsed state will fail. This change fixes
this issue by calling pm_runtime_get_sync() before ufshcd_hold_all().
Change-Id: Iff1d54d815efafbc17d56562de2e9065770c2612
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
During power mode change, PACP_PWR_Req frame sends
PAPowerModeUserData parameters (and they are considered valid by device if
Flags[4] - UserDataValid bit is set in the same frame).
Currently we don't set these PAPowerModeUserData parameters and hardware
always sets UserDataValid bit which would clear all the DL layer timeout
values of the peer device after the power mode change.
This change sets the PAPowerModeUserData[0..2] to UniPro specification
recommended default values, in addition we are also setting the relevant
DME_LOCAL_* timer attributes as required by UFS HCI specification.
Change-Id: I5ff98c03144e6973848d2fa1bf2aafd7637025fb
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
UFS clock scaling may do couple of things such as scaling UFS controller
clock frequencies and UFS interface gear scaling. To ensure that these
operations are completed successfully, driver need to make sure that there
are no regular transfer commands allowed until clock scaling is completed.
Currently we are only blocking the scsi command requests during gear
scaling but not during clock frequency scaling (and related operations).
So first fix is to block scsi requests for entire clock scaling operation,
we are doing this by invoking scsi_block_requests() and then waiting for
all pending requests (doorbells) to be completed. But this approach can not
always ensure that no new request will be issued while clock scaling is in
progress. Think of a scenario where one request just got queued via
->queuecommand() (but still haven't set the doorbell) when no doorbells
were pending and assume that clock scaling function has also started
executing in parallel. In this case clock scaling function blocks the new
scsi requests from being issued and wait for all the pending doorbells
to be cleared. As there were no doorbells pending when it checks for it,
it decides to go ahead with clock scaling but now the command with was just
issued via ->queuecommand() sets the doorbell and gets issued by
controller.
This change fixes such race condition by using the read-write semaphore
approach which basically blocks all the requests until clock scaling work
finishes but would allow multiple requests to be queued in parallel if
clock scaling work isn't running.
Change-Id: I99beed27bbc5b768fb3cee8b84cb5392619cace7
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
It seems arguments passed to memset function are swapped, this change
fixes it.
Change-Id: I1f69d528da9030a2bc061d3c9504058d8b71068b
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
We are only enabling the lane-0 symbol clock in 1-lane configuration
but it seems during disabling we are trying to disable lane-1 symbol
clocks as well even if they were never enable in 1-lane configuration,
this change fixes this minor issue.
Change-Id: I619a7c2e83a96caa338a56d53e61bba3d982ea13
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This patch adds a new debugfs capability to collect query statistics.
It counts how many times each IDN (index) was sent to device for each
query opcode.
A usage example:
1. to clean current query statistics:
echo 1 > /d/ufshcd0/stats/query_stats
2. to dump the current query statistics:
cat /d/ufshcd0/stats/query_stats
This capability is always on, and can not be turned off.
Change-Id: I46ec405aae480c0dc161dca015b407bde6335cf7
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
ufshcd_wait_for_doorbell_clr() function should wait for the
doorbell to be cleared until the specified timeout period but
current implementation really seems to be waiting until the doorbell
is cleared (instead of timing out after timeout period), this
change fixes this issue.
Change-Id: I0dee77db7d319250b5fab5f77e51175ea7d1d664
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
In this patch a new folder is created under the debugfs/ufshcd<x>
root folder, in which all statistics information is located and collected.
Change-Id: I3ea68a7c75b8252b6ad8487ff927e0fe520242e4
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
In case UFS_QCOM component is compiled as a module, there will be
some building errors. This change fixes them.
Also, since UFS_QCOM didn't have MODULE_LICENSE(), it failed to be
loaded as a module:
"ufs_qcom: module license 'unspecified' taints kernel."
This patch also fixes this error.
Change-Id: I96c7c1cf012b29c954e50e77ed0849a4255f39b3
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
This change fixes a possible access to null pointer variables.
Change-Id: I7b57883b71b0e8e8ca437c92cd6fd4f85db91310
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change fixes a compilation warning that happens if SCSI_UFS_QCOM
is compiled as a module.
Change-Id: I5c0272006f805f237e8acb922ddb4b3c15ff6b21
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
We put the UFS device in sleep state & UFS link in hibern8 state during
runtime suspaned. After this we put all the UFS rails in low power
modes immediately but it seems some devices may still draw more than sleep
current from UFS rails (especially from VCCQ rail) atleast for 500us.
To avoid this situation, this change adds 2ms delay before putting these
UFS rails in LPM mode.
Change-Id: Ifd8d3ef9e059f527613af0fcaa25ec2a282ad6c4
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Passing strings instead of enums for tracing events uses more of
the tracing buffer. Instead, pass the enum integer value and
convert to string when tracing event is formated using the
__print_symbolic() macro.
Change-Id: I91c4374fd079c6ea150373548975f2b259eb2d59
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
[subhashj@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
The following three trace events:
ufshcd_clk_gating, ufshcd_hibern8_on_idle and ufshcd_auto_bkops_state
share the same arguments and meaning - logging some state change
in the UFS driver.
Defining those as template instances takes up less memory compared
to be defined as separate trace events.
Change-Id: I92c2bf3eada6f876b8c9e8a7bfc4568c7886548f
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
This change adds a capability to record how many errors are
injected per error scenario, and also the debugfs entry to read them
and to reset them.
cat /d/ufshcd0/err_inj_stats
to analyze how many error were injected per error scenario
echo 1 > /d/ufshcd0/err_inj_stats
(or any other value, to reset all statistics)
Change-Id: Ide06c30d1a9d3b65df99cd8bc24cffecc9a4b724
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
In case error is actually being injected by the error injecting framework,
we would like to be able to tell the error code index as well as the
error code value.
So inspecting the debug messages and see error index would help
isolating a desired error code (via debugfs command) and testing
it specifically.
Change-Id: I624fd5ca55f21fe7c31a80e8ec2e4f8982b7f909
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds a debugfs capability to initiate a reset and restore
procedure to the UFS controller by executing:
echo 1 > /sys/kernel/debug/ufshcd0/reset_controller
This capability is necessary for testing purposes of the the reset
and restore procedure.
Change-Id: I126e2d80d2ac63be6107aee29fa03dce4f56e83a
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds interrupt error injection.
It also modifies the logic of interrupt error injection scenario
to support the UFS fault injection generic framework.
Change-Id: Ibdcd4f875fccbc359a92c71d088a06440fe22082
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds a few debugfs options for the UFS fault injection
framework.
"err_inj_scenario" entry - to enable/disable error scenarios.
"err_inj_codes" entry - to control specific error codes for a specific
error scenario.
usage:
echo 0x5 > /sys/kernel/debug/ufshcd0/err_inj_scenario
as 0x5 is b0101, it sets bits #0 and #2 of err_inj_scenario_mask.
echo "2, 0x7" > /sys/kernel/debug/ufshcd0/err_inj_codes
sets error codes b0111 for error scenario #2
Change-Id: I0c58886a1ffac4dc85d8827bf69d650082a03fc5
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
[subhashj@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
If both host and device support UniPro Specification v1.6 then many of the
important UniPro timing parameters are auto tuned after the link start up.
But due to SW bug these auto tuned parameters might get overwritten, this
change fixes the issue by properly checking the UniPro specifcation
version.
Change-Id: I152d5dcfac0bc0f8a5dc6b1e7267e254ec0d5d80
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Few Toshiba UFS device models advertise RX_MIN_ACTIVATETIME_CAPABILITY as
600us which may not be enough for reliable hibern8 exit hardware sequence
from UFS device.
To workaround this issue, host should set its PA_TACTIVATE time to 1ms even
if device advertises RX_MIN_ACTIVATETIME_CAPABILITY less than 1ms.
Change-Id: I91d0fcdecbcd3294a6a67556b5bf1c7d2636cff1
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Perform PHY and controller hard reset to recover from
hibern8 exit failure.
This requires full initialization of the PHY and the controller
before issuing link start-up.
Change-Id: I93c5f896d86eb74a1ef490e3e14ae796082888cf
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
