Add debug-fs entry to show current state of UFS PM QoS.
Change-Id: Ic1ae582b5a657f332be8a6f8d3bbcc0434bdf3d3
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Add sys-fs entries to allow user space control PM QoS
latency parameters and enable/disable PM QoS voting.
The entries are added under:
/sys/bus/platform/devices/624000.ufshc/624000.ufshc:ufs_variant/
pm_qos_enable:
write 0 to disable PM QoS, 1 to enable.
Example: "echo 1 > pm_qos_enable"
pm_qos_latency_us:
write the desired value for each cpu group, separated by a comma.
Example: "echo 10,20 > pm_qos_latency_us"
Change-Id: I9797a1e62c4867ab831b4f18cbb1e0ca9834247b
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Signed-off-by: Krishna Konda <kkonda@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Current UFS PM QoS design and implementation do not seem to give
the desired increase in performance. This change revisits the
PM QoS implementation trying to improve performance by making
the following changes:
* de-couple voting from clock scaling decision so voting
occurs from the first request and unvoting on the completion
of the last request regardless to clock scaling state.
Otherwise, suspending the PM QoS voting during the time it takes
to decide on clock up-scaling, seems to degrade random access
performance.
* vote on a per-cluster basis by inspecting the request object's
cpu field. This follows the soft-irq allocation scheme in the
block layer, so the cpu to which the block layer schedules the
soft-irq will not be put into deep LPM.
We should note that PM QoS voting using cpu mask for specific
cpus is a feature of the qcom specific PM QoS implementation.
Change-Id: I427d202aeb45cd284a3bb128e26e519212614801
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Signed-off-by: Krishna Konda <kkonda@codeaurora.org>
[venkatg@codeaurora.org: resolved merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
The new UFS Controller (version 3.0.0) has a few changes compared to
previous versions, such as a new vendor specific register, and
different debug register offsets.
Change-Id: Idf455d2dcbdcdadcb1e809a8349fc9adbe9733dd
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
Some platforms do not enable the BUS_SCALING feature.
For such cases, we should wrap the routines that use this feature
with a configuration flag.
Change-Id: Iced5b6492a470eb5700b50a50de8260b6bb59753
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
The UFS HCI v2.1 includes a few additional registers. This change
updates the HCI register, the UFS version register content and
the Interrupt Status register.
Change-Id: Icfeef0e6aaed064b456fac1cd5531fff1328ae37
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
When we are resuming the UFS device rails in HPM mode, we are first
powering on the VCC rail while VCCQ and VCCQ2 rails still being in LPM
mode. Some UFS devices may take VCC on event as hint that host wants
UFS device to be resumed and may start drawing more power from the
VCCQ/VCCQ2 rails (while they are still in LPM mode) causing voltage drop
on these rails. This change fixes this issue by bringing VCCQ & VCCQ2 rails
out of LPM before powering on VCC rail.
Change-Id: I950591d2a571257e08d7697e2b33dd5968f7333f
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This patch splits the vops by type and creates wrapper
functions for all vops.
Used in order clarify the runtime code and to allow
further types of vops.
This is the first step towards rearchitecting the
variant implementation of the UFS driver - per requirement
from the community. As this will become a full-blown
rearchitecture - incrementing the driver version.
Change-Id: Ifd0a5412a608590bc6c761caf0b3a87aed37f70b
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
[venkatg@codeaurora.org: resolved merge conflicts,
change ufshcd_pltfrm_init to use ufs_hba_variant
instead of ufs_hba_variant_ops]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Adds setter/getter functions to set/get a pointer
to the variant control structure in/from the generic
hba control structure.
Change-Id: I4c7b3967366f394472490410e3880fdc6e3608b4
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
[venkatg@codeaurora.org: use ufshcd_set_variant instead of
ufshcd_bind_variant]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
When compiling with gcc5, it throws below errors:
drivers/scsi/ufs/ufs-qcom.c: In function 'ufs_qcom_hce_enable_notify':
drivers/scsi/ufs/ufs-qcom.c:352:10: warning: switch condition has boolean
value [-Wswitch-bool] error, forbidden warning: ufs-qcom.c:352
This is because we are using the boolean value in switch condition and it
seems to be forbidden by compiler (by default). Fix this by using the
proper enum type instead of boolean.
Change-Id: Iee6950e5986dc51afbf9440cd4773b4d0f02fcdc
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflict]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
We are currently running UFS link in HS-G3 FAST mode during high load
condition for best possible performance and in HS-G2 FAST mode during
low load condition to save power. As we are anyway scaling down from
HS-G3 to HS-G2, we can also change the mode from FAST to FASTAUTO.
So we looked at the performance numbers with HS-G2 FASTAUTO mode and
they are good enough for most of the low bandwidth usecases. But Samsung
UFS memory devices are exception which has really low sequential read
throughput in FAST AUTO mode hence we will only be enabling FAST AUTO mode
for other UFS device vendors.
Change-Id: Ia79da3e82fb87c0e396534979398617388f40af9
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
DME commands such as Hibern8 enter/exit and gear switch generate 2
completion interrupts, one for confirmation that command is received
by local UniPro and 2nd one is the final confirmation after communication
with remote UniPro. Currently both of these completions are registered
as interrupt events which is not quite necessary and instead we can
just wait for the interrupt of 2nd completion, this should reduce
the number of interrupts and could reduce the unnecessary CPU wakeups
to handle extra interrupts.
Change-Id: I5365b8b5cae952fd4bd7bb93cca0cfe76ea8b619
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
UniPro v1.6 specication requires both host and device TX LCC Enable to be
in same state but current driver still attempts to disable just one
side of LCC. With this change, if Host supports UniPro v1.6 (and all
commercial UFS devices anyway supports the UniPro v1.6), we would keep
both host and device TX LCC disabled after the link startup. Keeping LCC
disabled helps avoid multiple compatibility issues between host and device.
Change-Id: I1697089255b63f70a465c8b42a899ee6ab0acd88
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
DME attributes read/write commands need GenSelectorIndex argument and
according to UFS host controller specification, it should specify the
targeted M-PHY data lane.
This is the valid range of GenSelectorIndex for M-PHY attributes:
0 to (2 * PA_MaxDataLanes - 1)
Example (Note: PA_MaxDataLanes is UniPro protocol constant set to 4):
-----------------------------------------
MPHY Access target | GenSelectorIndex
-----------------------------------------
TX Lane0 | 0
TX Lane1 | 1
RX Lane0 | 4
RX Lane1 | 5
-----------------------------------------
This change makes sure that correct GenSelectorIndex is passed for M-PHY
attributes while it reading via debugfs.
Change-Id: If09e4271958e4bb18d315446cef253caec75584a
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Quite a few UFS functions (especially those which are invoked in debugfs
context) are calling the ufshcd_hold_all() first before runtime resuming
the UFS host controller. As runtime suspend might have already power
collapsed UFS core controller, attempt to enable clocks (as done by
ufshcd_hold_all()) in power collapsed state will fail. This change fixes
this issue by calling pm_runtime_get_sync() before ufshcd_hold_all().
Change-Id: Iff1d54d815efafbc17d56562de2e9065770c2612
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
During power mode change, PACP_PWR_Req frame sends
PAPowerModeUserData parameters (and they are considered valid by device if
Flags[4] - UserDataValid bit is set in the same frame).
Currently we don't set these PAPowerModeUserData parameters and hardware
always sets UserDataValid bit which would clear all the DL layer timeout
values of the peer device after the power mode change.
This change sets the PAPowerModeUserData[0..2] to UniPro specification
recommended default values, in addition we are also setting the relevant
DME_LOCAL_* timer attributes as required by UFS HCI specification.
Change-Id: I5ff98c03144e6973848d2fa1bf2aafd7637025fb
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
UFS clock scaling may do couple of things such as scaling UFS controller
clock frequencies and UFS interface gear scaling. To ensure that these
operations are completed successfully, driver need to make sure that there
are no regular transfer commands allowed until clock scaling is completed.
Currently we are only blocking the scsi command requests during gear
scaling but not during clock frequency scaling (and related operations).
So first fix is to block scsi requests for entire clock scaling operation,
we are doing this by invoking scsi_block_requests() and then waiting for
all pending requests (doorbells) to be completed. But this approach can not
always ensure that no new request will be issued while clock scaling is in
progress. Think of a scenario where one request just got queued via
->queuecommand() (but still haven't set the doorbell) when no doorbells
were pending and assume that clock scaling function has also started
executing in parallel. In this case clock scaling function blocks the new
scsi requests from being issued and wait for all the pending doorbells
to be cleared. As there were no doorbells pending when it checks for it,
it decides to go ahead with clock scaling but now the command with was just
issued via ->queuecommand() sets the doorbell and gets issued by
controller.
This change fixes such race condition by using the read-write semaphore
approach which basically blocks all the requests until clock scaling work
finishes but would allow multiple requests to be queued in parallel if
clock scaling work isn't running.
Change-Id: I99beed27bbc5b768fb3cee8b84cb5392619cace7
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
It seems arguments passed to memset function are swapped, this change
fixes it.
Change-Id: I1f69d528da9030a2bc061d3c9504058d8b71068b
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
We are only enabling the lane-0 symbol clock in 1-lane configuration
but it seems during disabling we are trying to disable lane-1 symbol
clocks as well even if they were never enable in 1-lane configuration,
this change fixes this minor issue.
Change-Id: I619a7c2e83a96caa338a56d53e61bba3d982ea13
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This patch adds a new debugfs capability to collect query statistics.
It counts how many times each IDN (index) was sent to device for each
query opcode.
A usage example:
1. to clean current query statistics:
echo 1 > /d/ufshcd0/stats/query_stats
2. to dump the current query statistics:
cat /d/ufshcd0/stats/query_stats
This capability is always on, and can not be turned off.
Change-Id: I46ec405aae480c0dc161dca015b407bde6335cf7
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
ufshcd_wait_for_doorbell_clr() function should wait for the
doorbell to be cleared until the specified timeout period but
current implementation really seems to be waiting until the doorbell
is cleared (instead of timing out after timeout period), this
change fixes this issue.
Change-Id: I0dee77db7d319250b5fab5f77e51175ea7d1d664
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
In this patch a new folder is created under the debugfs/ufshcd<x>
root folder, in which all statistics information is located and collected.
Change-Id: I3ea68a7c75b8252b6ad8487ff927e0fe520242e4
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
In case UFS_QCOM component is compiled as a module, there will be
some building errors. This change fixes them.
Also, since UFS_QCOM didn't have MODULE_LICENSE(), it failed to be
loaded as a module:
"ufs_qcom: module license 'unspecified' taints kernel."
This patch also fixes this error.
Change-Id: I96c7c1cf012b29c954e50e77ed0849a4255f39b3
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
This change fixes a possible access to null pointer variables.
Change-Id: I7b57883b71b0e8e8ca437c92cd6fd4f85db91310
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change fixes a compilation warning that happens if SCSI_UFS_QCOM
is compiled as a module.
Change-Id: I5c0272006f805f237e8acb922ddb4b3c15ff6b21
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
We put the UFS device in sleep state & UFS link in hibern8 state during
runtime suspaned. After this we put all the UFS rails in low power
modes immediately but it seems some devices may still draw more than sleep
current from UFS rails (especially from VCCQ rail) atleast for 500us.
To avoid this situation, this change adds 2ms delay before putting these
UFS rails in LPM mode.
Change-Id: Ifd8d3ef9e059f527613af0fcaa25ec2a282ad6c4
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Passing strings instead of enums for tracing events uses more of
the tracing buffer. Instead, pass the enum integer value and
convert to string when tracing event is formated using the
__print_symbolic() macro.
Change-Id: I91c4374fd079c6ea150373548975f2b259eb2d59
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
[subhashj@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
The following three trace events:
ufshcd_clk_gating, ufshcd_hibern8_on_idle and ufshcd_auto_bkops_state
share the same arguments and meaning - logging some state change
in the UFS driver.
Defining those as template instances takes up less memory compared
to be defined as separate trace events.
Change-Id: I92c2bf3eada6f876b8c9e8a7bfc4568c7886548f
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
This change adds a capability to record how many errors are
injected per error scenario, and also the debugfs entry to read them
and to reset them.
cat /d/ufshcd0/err_inj_stats
to analyze how many error were injected per error scenario
echo 1 > /d/ufshcd0/err_inj_stats
(or any other value, to reset all statistics)
Change-Id: Ide06c30d1a9d3b65df99cd8bc24cffecc9a4b724
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
In case error is actually being injected by the error injecting framework,
we would like to be able to tell the error code index as well as the
error code value.
So inspecting the debug messages and see error index would help
isolating a desired error code (via debugfs command) and testing
it specifically.
Change-Id: I624fd5ca55f21fe7c31a80e8ec2e4f8982b7f909
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds a debugfs capability to initiate a reset and restore
procedure to the UFS controller by executing:
echo 1 > /sys/kernel/debug/ufshcd0/reset_controller
This capability is necessary for testing purposes of the the reset
and restore procedure.
Change-Id: I126e2d80d2ac63be6107aee29fa03dce4f56e83a
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds interrupt error injection.
It also modifies the logic of interrupt error injection scenario
to support the UFS fault injection generic framework.
Change-Id: Ibdcd4f875fccbc359a92c71d088a06440fe22082
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds a few debugfs options for the UFS fault injection
framework.
"err_inj_scenario" entry - to enable/disable error scenarios.
"err_inj_codes" entry - to control specific error codes for a specific
error scenario.
usage:
echo 0x5 > /sys/kernel/debug/ufshcd0/err_inj_scenario
as 0x5 is b0101, it sets bits #0 and #2 of err_inj_scenario_mask.
echo "2, 0x7" > /sys/kernel/debug/ufshcd0/err_inj_codes
sets error codes b0111 for error scenario #2
Change-Id: I0c58886a1ffac4dc85d8827bf69d650082a03fc5
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
[subhashj@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
If both host and device support UniPro Specification v1.6 then many of the
important UniPro timing parameters are auto tuned after the link start up.
But due to SW bug these auto tuned parameters might get overwritten, this
change fixes the issue by properly checking the UniPro specifcation
version.
Change-Id: I152d5dcfac0bc0f8a5dc6b1e7267e254ec0d5d80
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Few Toshiba UFS device models advertise RX_MIN_ACTIVATETIME_CAPABILITY as
600us which may not be enough for reliable hibern8 exit hardware sequence
from UFS device.
To workaround this issue, host should set its PA_TACTIVATE time to 1ms even
if device advertises RX_MIN_ACTIVATETIME_CAPABILITY less than 1ms.
Change-Id: I91d0fcdecbcd3294a6a67556b5bf1c7d2636cff1
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Perform PHY and controller hard reset to recover from
hibern8 exit failure.
This requires full initialization of the PHY and the controller
before issuing link start-up.
Change-Id: I93c5f896d86eb74a1ef490e3e14ae796082888cf
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This change fixes the compilation errors seen when DEBUG_FS is disabled.
Change-Id: I15ba780ddfd3f6238bdd0434f1769c2986ec832f
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
We are seeing that some devices are raising the urgent bkops exception
events even when BKOPS status doesn't indicate performace impacted or
critical. Handle these device by determining their urgent bkops status
at runtime.
Change-Id: I0392c6d1f1ca820026906dac91beba7c8719edc8
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Add range check on tag statistics array index to avoid a potential
situation where the array index is out-of-bounds in case
TS_NOT_SUPPORTED is returned as the index.
Change-Id: Ib35fd0abe9553f6ea259efd2f98c7a4ef490d6ed
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
When ufshcd_host_reset_and_restore() is called, it is possible
that ufshcd_probe_hba() fails and return an error value.
However, the later call to crypto_engine_reset() might succeed
and override the previous error value.
This will result in ufshcd_host_reset_and_restore() returning
success while in fact a serious error may have occured which
should prevent us from continuing in the normal sequence.
Instead, call crypto_engine_reset() only if ufshcd_probe_hba()
was successfull.
Change-Id: I0462f674b8750bfb9e0657cb37833aa2bd2aa4f0
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
Commit 4f98a9564b745f3b0b1bea02d351c9f8f2f17d39 (scsi: ufs: fix race
between clock gating and scaling work) tried to fix the race condition
between clock gating and scaling work but it didn't fixed the race in
all possible states. This change fixes the race condition by making
sure that we hold the clock reference even if clocks are ON when we
entered into clock scaling work.
Change-Id: Ic3cf9224f5afb2900fe2553ce8c302cc8b20e623
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
We print the ERROR level log message for every gear switch which would
generally end up in serial console but clock scaling also does gear
switch and these gear switch log messages might clutter the kernel logs
unnecessarily hence this change avoids printing gear switch log messages
during clock scaling.
Change-Id: If1079bbff633c3270ddae8579801d1f265707254
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This reverts commit a08f72e5e1453acac1d3fd50a4ed314ea5ff643c. This
commit was added to workaround the issue seen with particular UFS device
vendor. As this issue is already fixed by UFS device vendor, we no longer
needs this workaround hence reverting the original patch.
Change-Id: Ib2fd45a893042053eee18fe93554e7b02c4732c6
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Clock scaling polling time window is currently 100ms which is more than
the clock gating time which is 50ms and during clock gating, clock scaling
logic would be disabled. This may make the clocks to remain scaled down
for longer period of time, fix this by reducing the clock scaling polling
window less than clock gating timeout.
Change-Id: I8365470c7a6bd4950c1decbd418272359475c366
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This reverts commit 5f84d7bb3056bd62e6bcc7b6b9801436a49d9a52.
There was possible race condition between clock scaling and clock gating
work hence clock scaling was disabled until proper fix for the race
condition is found. Now that race condition is properly fixed, this commit
is reverted back.
Change-Id: I1d2e7be89cd5bb36c37ffe26a1f2606c0800c434
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
UFS Clock scaling work may change the UFS gear mode as well which requires
holding the vote for UFS clocks but if UFS clock gating work is running
at the same time, it may end up waiting for UFS clock scaling work to
finish hence enters into deadlock state. Here is the call stack for this
deadlock state:
Clock scaling work:
__schedule()
schedule()
schedule_timeout()
do_wait_for_common(inline)
__wait_for_common(inline)
wait_for_common()
wait_for_completion()
flush_work()
ufshcd_hold()
ufshcd_pm_qos_hold(inline)
ufshcd_hold_all()
ufshcd_wait_for_doorbell_clr()
ufshcd_scale_gear()
ufshcd_devfreq_scale(inline)
ufshcd_devfreq_target()
update_devfreq()
devfreq_monitor()
static_key_false(inline)
trace_workqueue_execute_end(inline)
process_one_work()
worker_thread()
kthread()
Clock gating work:
__schedule()
schedule()
schedule_preempt_disabled()
__mutex_lock_common(inline)
__mutex_lock_slowpath()
current_thread_info(inline)
mutex_set_owner(inline)
mutex_lock()
devfreq_monitor_resume()
devfreq_simple_ondemand_handler()
devfreq_resume_device()
ufshcd_ungate_work()
static_key_false(inline)
trace_workqueue_execute_end(inline)
process_one_work()
process_scheduled_works(inline)
worker_thread()
kthread()
This change avoids this deadlock by cancelling the clock gating work before
voting for clocks in clock scaling work.
Change-Id: Icae8762282fc10032ae9bb823ab9f5de9912b2e5
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
spin_lock_irq() / spin_unlock_irq() is used so interrupts are
enabled after unlocking the spinlock. However, it is not guaranteed
they were enabled before.
This change uses the proper irqsave / irqrestore variants instead.
Without it, a spinlock recursion on the scsi request completion path
is possible if completion interrupt occurs.
Change-Id: If5a001f195466f85e50df6ce23fd1e10851c5e2b
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
