Quite a few UFS functions (especially those which are invoked in debugfs
context) are calling the ufshcd_hold_all() first before runtime resuming
the UFS host controller. As runtime suspend might have already power
collapsed UFS core controller, attempt to enable clocks (as done by
ufshcd_hold_all()) in power collapsed state will fail. This change fixes
this issue by calling pm_runtime_get_sync() before ufshcd_hold_all().
Change-Id: Iff1d54d815efafbc17d56562de2e9065770c2612
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
During power mode change, PACP_PWR_Req frame sends
PAPowerModeUserData parameters (and they are considered valid by device if
Flags[4] - UserDataValid bit is set in the same frame).
Currently we don't set these PAPowerModeUserData parameters and hardware
always sets UserDataValid bit which would clear all the DL layer timeout
values of the peer device after the power mode change.
This change sets the PAPowerModeUserData[0..2] to UniPro specification
recommended default values, in addition we are also setting the relevant
DME_LOCAL_* timer attributes as required by UFS HCI specification.
Change-Id: I5ff98c03144e6973848d2fa1bf2aafd7637025fb
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
UFS clock scaling may do couple of things such as scaling UFS controller
clock frequencies and UFS interface gear scaling. To ensure that these
operations are completed successfully, driver need to make sure that there
are no regular transfer commands allowed until clock scaling is completed.
Currently we are only blocking the scsi command requests during gear
scaling but not during clock frequency scaling (and related operations).
So first fix is to block scsi requests for entire clock scaling operation,
we are doing this by invoking scsi_block_requests() and then waiting for
all pending requests (doorbells) to be completed. But this approach can not
always ensure that no new request will be issued while clock scaling is in
progress. Think of a scenario where one request just got queued via
->queuecommand() (but still haven't set the doorbell) when no doorbells
were pending and assume that clock scaling function has also started
executing in parallel. In this case clock scaling function blocks the new
scsi requests from being issued and wait for all the pending doorbells
to be cleared. As there were no doorbells pending when it checks for it,
it decides to go ahead with clock scaling but now the command with was just
issued via ->queuecommand() sets the doorbell and gets issued by
controller.
This change fixes such race condition by using the read-write semaphore
approach which basically blocks all the requests until clock scaling work
finishes but would allow multiple requests to be queued in parallel if
clock scaling work isn't running.
Change-Id: I99beed27bbc5b768fb3cee8b84cb5392619cace7
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
It seems arguments passed to memset function are swapped, this change
fixes it.
Change-Id: I1f69d528da9030a2bc061d3c9504058d8b71068b
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
We are only enabling the lane-0 symbol clock in 1-lane configuration
but it seems during disabling we are trying to disable lane-1 symbol
clocks as well even if they were never enable in 1-lane configuration,
this change fixes this minor issue.
Change-Id: I619a7c2e83a96caa338a56d53e61bba3d982ea13
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This patch adds a new debugfs capability to collect query statistics.
It counts how many times each IDN (index) was sent to device for each
query opcode.
A usage example:
1. to clean current query statistics:
echo 1 > /d/ufshcd0/stats/query_stats
2. to dump the current query statistics:
cat /d/ufshcd0/stats/query_stats
This capability is always on, and can not be turned off.
Change-Id: I46ec405aae480c0dc161dca015b407bde6335cf7
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
ufshcd_wait_for_doorbell_clr() function should wait for the
doorbell to be cleared until the specified timeout period but
current implementation really seems to be waiting until the doorbell
is cleared (instead of timing out after timeout period), this
change fixes this issue.
Change-Id: I0dee77db7d319250b5fab5f77e51175ea7d1d664
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
In this patch a new folder is created under the debugfs/ufshcd<x>
root folder, in which all statistics information is located and collected.
Change-Id: I3ea68a7c75b8252b6ad8487ff927e0fe520242e4
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
In case UFS_QCOM component is compiled as a module, there will be
some building errors. This change fixes them.
Also, since UFS_QCOM didn't have MODULE_LICENSE(), it failed to be
loaded as a module:
"ufs_qcom: module license 'unspecified' taints kernel."
This patch also fixes this error.
Change-Id: I96c7c1cf012b29c954e50e77ed0849a4255f39b3
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
This change fixes a possible access to null pointer variables.
Change-Id: I7b57883b71b0e8e8ca437c92cd6fd4f85db91310
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change fixes a compilation warning that happens if SCSI_UFS_QCOM
is compiled as a module.
Change-Id: I5c0272006f805f237e8acb922ddb4b3c15ff6b21
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
We put the UFS device in sleep state & UFS link in hibern8 state during
runtime suspaned. After this we put all the UFS rails in low power
modes immediately but it seems some devices may still draw more than sleep
current from UFS rails (especially from VCCQ rail) atleast for 500us.
To avoid this situation, this change adds 2ms delay before putting these
UFS rails in LPM mode.
Change-Id: Ifd8d3ef9e059f527613af0fcaa25ec2a282ad6c4
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Passing strings instead of enums for tracing events uses more of
the tracing buffer. Instead, pass the enum integer value and
convert to string when tracing event is formated using the
__print_symbolic() macro.
Change-Id: I91c4374fd079c6ea150373548975f2b259eb2d59
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
[subhashj@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
The following three trace events:
ufshcd_clk_gating, ufshcd_hibern8_on_idle and ufshcd_auto_bkops_state
share the same arguments and meaning - logging some state change
in the UFS driver.
Defining those as template instances takes up less memory compared
to be defined as separate trace events.
Change-Id: I92c2bf3eada6f876b8c9e8a7bfc4568c7886548f
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
This change adds a capability to record how many errors are
injected per error scenario, and also the debugfs entry to read them
and to reset them.
cat /d/ufshcd0/err_inj_stats
to analyze how many error were injected per error scenario
echo 1 > /d/ufshcd0/err_inj_stats
(or any other value, to reset all statistics)
Change-Id: Ide06c30d1a9d3b65df99cd8bc24cffecc9a4b724
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
In case error is actually being injected by the error injecting framework,
we would like to be able to tell the error code index as well as the
error code value.
So inspecting the debug messages and see error index would help
isolating a desired error code (via debugfs command) and testing
it specifically.
Change-Id: I624fd5ca55f21fe7c31a80e8ec2e4f8982b7f909
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds a debugfs capability to initiate a reset and restore
procedure to the UFS controller by executing:
echo 1 > /sys/kernel/debug/ufshcd0/reset_controller
This capability is necessary for testing purposes of the the reset
and restore procedure.
Change-Id: I126e2d80d2ac63be6107aee29fa03dce4f56e83a
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds interrupt error injection.
It also modifies the logic of interrupt error injection scenario
to support the UFS fault injection generic framework.
Change-Id: Ibdcd4f875fccbc359a92c71d088a06440fe22082
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
This change adds a few debugfs options for the UFS fault injection
framework.
"err_inj_scenario" entry - to enable/disable error scenarios.
"err_inj_codes" entry - to control specific error codes for a specific
error scenario.
usage:
echo 0x5 > /sys/kernel/debug/ufshcd0/err_inj_scenario
as 0x5 is b0101, it sets bits #0 and #2 of err_inj_scenario_mask.
echo "2, 0x7" > /sys/kernel/debug/ufshcd0/err_inj_codes
sets error codes b0111 for error scenario #2
Change-Id: I0c58886a1ffac4dc85d8827bf69d650082a03fc5
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
[subhashj@codeaurora.org: resolved trivial merge conflicts]
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
If both host and device support UniPro Specification v1.6 then many of the
important UniPro timing parameters are auto tuned after the link start up.
But due to SW bug these auto tuned parameters might get overwritten, this
change fixes the issue by properly checking the UniPro specifcation
version.
Change-Id: I152d5dcfac0bc0f8a5dc6b1e7267e254ec0d5d80
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Few Toshiba UFS device models advertise RX_MIN_ACTIVATETIME_CAPABILITY as
600us which may not be enough for reliable hibern8 exit hardware sequence
from UFS device.
To workaround this issue, host should set its PA_TACTIVATE time to 1ms even
if device advertises RX_MIN_ACTIVATETIME_CAPABILITY less than 1ms.
Change-Id: I91d0fcdecbcd3294a6a67556b5bf1c7d2636cff1
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Perform PHY and controller hard reset to recover from
hibern8 exit failure.
This requires full initialization of the PHY and the controller
before issuing link start-up.
Change-Id: I93c5f896d86eb74a1ef490e3e14ae796082888cf
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This change fixes the compilation errors seen when DEBUG_FS is disabled.
Change-Id: I15ba780ddfd3f6238bdd0434f1769c2986ec832f
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
We are seeing that some devices are raising the urgent bkops exception
events even when BKOPS status doesn't indicate performace impacted or
critical. Handle these device by determining their urgent bkops status
at runtime.
Change-Id: I0392c6d1f1ca820026906dac91beba7c8719edc8
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Add range check on tag statistics array index to avoid a potential
situation where the array index is out-of-bounds in case
TS_NOT_SUPPORTED is returned as the index.
Change-Id: Ib35fd0abe9553f6ea259efd2f98c7a4ef490d6ed
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
When ufshcd_host_reset_and_restore() is called, it is possible
that ufshcd_probe_hba() fails and return an error value.
However, the later call to crypto_engine_reset() might succeed
and override the previous error value.
This will result in ufshcd_host_reset_and_restore() returning
success while in fact a serious error may have occured which
should prevent us from continuing in the normal sequence.
Instead, call crypto_engine_reset() only if ufshcd_probe_hba()
was successfull.
Change-Id: I0462f674b8750bfb9e0657cb37833aa2bd2aa4f0
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Signed-off-by: Dov Levenglick <dovl@codeaurora.org>
Commit 4f98a9564b745f3b0b1bea02d351c9f8f2f17d39 (scsi: ufs: fix race
between clock gating and scaling work) tried to fix the race condition
between clock gating and scaling work but it didn't fixed the race in
all possible states. This change fixes the race condition by making
sure that we hold the clock reference even if clocks are ON when we
entered into clock scaling work.
Change-Id: Ic3cf9224f5afb2900fe2553ce8c302cc8b20e623
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
We print the ERROR level log message for every gear switch which would
generally end up in serial console but clock scaling also does gear
switch and these gear switch log messages might clutter the kernel logs
unnecessarily hence this change avoids printing gear switch log messages
during clock scaling.
Change-Id: If1079bbff633c3270ddae8579801d1f265707254
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This reverts commit a08f72e5e1453acac1d3fd50a4ed314ea5ff643c. This
commit was added to workaround the issue seen with particular UFS device
vendor. As this issue is already fixed by UFS device vendor, we no longer
needs this workaround hence reverting the original patch.
Change-Id: Ib2fd45a893042053eee18fe93554e7b02c4732c6
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
Clock scaling polling time window is currently 100ms which is more than
the clock gating time which is 50ms and during clock gating, clock scaling
logic would be disabled. This may make the clocks to remain scaled down
for longer period of time, fix this by reducing the clock scaling polling
window less than clock gating timeout.
Change-Id: I8365470c7a6bd4950c1decbd418272359475c366
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
This reverts commit 5f84d7bb3056bd62e6bcc7b6b9801436a49d9a52.
There was possible race condition between clock scaling and clock gating
work hence clock scaling was disabled until proper fix for the race
condition is found. Now that race condition is properly fixed, this commit
is reverted back.
Change-Id: I1d2e7be89cd5bb36c37ffe26a1f2606c0800c434
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
UFS Clock scaling work may change the UFS gear mode as well which requires
holding the vote for UFS clocks but if UFS clock gating work is running
at the same time, it may end up waiting for UFS clock scaling work to
finish hence enters into deadlock state. Here is the call stack for this
deadlock state:
Clock scaling work:
__schedule()
schedule()
schedule_timeout()
do_wait_for_common(inline)
__wait_for_common(inline)
wait_for_common()
wait_for_completion()
flush_work()
ufshcd_hold()
ufshcd_pm_qos_hold(inline)
ufshcd_hold_all()
ufshcd_wait_for_doorbell_clr()
ufshcd_scale_gear()
ufshcd_devfreq_scale(inline)
ufshcd_devfreq_target()
update_devfreq()
devfreq_monitor()
static_key_false(inline)
trace_workqueue_execute_end(inline)
process_one_work()
worker_thread()
kthread()
Clock gating work:
__schedule()
schedule()
schedule_preempt_disabled()
__mutex_lock_common(inline)
__mutex_lock_slowpath()
current_thread_info(inline)
mutex_set_owner(inline)
mutex_lock()
devfreq_monitor_resume()
devfreq_simple_ondemand_handler()
devfreq_resume_device()
ufshcd_ungate_work()
static_key_false(inline)
trace_workqueue_execute_end(inline)
process_one_work()
process_scheduled_works(inline)
worker_thread()
kthread()
This change avoids this deadlock by cancelling the clock gating work before
voting for clocks in clock scaling work.
Change-Id: Icae8762282fc10032ae9bb823ab9f5de9912b2e5
Signed-off-by: Subhash Jadavani <subhashj@codeaurora.org>
spin_lock_irq() / spin_unlock_irq() is used so interrupts are
enabled after unlocking the spinlock. However, it is not guaranteed
they were enabled before.
This change uses the proper irqsave / irqrestore variants instead.
Without it, a spinlock recursion on the scsi request completion path
is possible if completion interrupt occurs.
Change-Id: If5a001f195466f85e50df6ce23fd1e10851c5e2b
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Since "major", "minor" and "step" are now being saved in
struct ufs_qcom_host, there is no need to call
ufs_qcom_get_controller_revision() here. In addition, those local
variables are not in use in this routine.
Change-Id: I2606f87fc5e54b45a86240c1bccdd8c47009e11a
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
Instead of assigning a value to the variable pointed to by
ice_status, the ice_status pointer is changed to NULL. This
is incorrect and this change fixes that error.
Change-Id: I10516635c73cd459341991a4c6f11c0ed6bda71f
Signed-off-by: Krishna Konda <kkonda@codeaurora.org>
This change extends the UFS fault injection framework.
It defines multiple fault injection scenarios, and data structures
that hold optional error codes for each error scenario.
When error is being injected, an error code is being randomly chosen
from the enabled error codes.
Change-Id: Id844c765f9fd3590f86f9ae0efdb43d275bdb646
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
Use the user-defined sector_range instead of the previously hard-coded
TEST_MAX_SECTOR_RANGE. If the user does not supply a sector_range, this
variable will default to 512MiB.
For the long sequential tests, sector_range will define the size of the
sequential I/O to submit. For the long random tests, sector_range will
define the range in which to submit random I/O as well as the total size
of the random I/O which is defined by the combination of sector_range and
LONG_RAND_TEST_REQ_RATIO.
Change-Id: Ifc7332e6def75c49448aadbebd35b7b9b3903447
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Unit tests submit large requests of 512KB made of 128 bios.
Current allocation was done via kmalloc which may not be able
to allocate such a large buffer which is also physically contiguous.
Using kmalloc to allocate each bio separately is also problematic as
it might not be page aligned. Some bio may end up using more than a
single memory segment which will fail the mapping of the bios to
the request which supports up to 128 physical segments.
To avoid such failure, allocate a separate page for each bio
(bio size is single page size).
Change-Id: Id0da394d458942e093d924bc7aa23aa3231cdca7
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
[venkatg@codeaurora.org: dropped block/test-iosched and mmc_test
related changes]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Current test-iosched design enables running only a single test
for a single block device.
This change modifies the test-iosched framework to allow running
several tests on several block devices.
Change-Id: I051d842733873488b64e89053d9c4e30e1249870
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
[merez@codeaurora.org: fix conflicts due to removal of BKOPs UT]
Signed-off-by: Maya Erez <merez@codeaurora.org>
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
[venkatg@codeaurora.org: dropped block/test-iosched and mmc_test
related changes]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Multi-query test fails on write descriptor requests becuase it
tries to write to the unit descriptor which is read-only according
to the spec.
The only writeable descriptors are the configuration and OEM ID
string descriptors which will be non-trivial to use.
Avoid sending write descriptor query for the time being, and replace
it with a read descriptor operation.
Change-Id: Ic121d9ff20f144e724df5436f9dbd44cde72e97d
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
It was found that the UFS unit tests fail when they are run on an external
UFS device. This happens when the external UFS device is suspended while
running the unit-tests, and the un-natural flow of test-iosched does
not resume the UFS driver and device in the correct manner.
We solve this by using the block layer power management API for running
the request queue, as well as explicitly resuming the device before
running the unit-tests.
Change-Id: I8273d4dd4f32e91bad28c019c159696cf8fabb42
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
This testcase checks data integrity in a random I/O usecase.
The test writes QUEUE_MAX_REQUESTS (usually 118) requests of size 4KB
to randomly and uniquely chosen LBAs. A different pattern is written
to each LBA (pattern written is index of LBA). After successfully writing,
the test reads from the above LBAs and verifies the pattern.
Change-Id: I09c6dda12834f06ae393411501f443bfd058dea4
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
The UFS tests are used for testing the functionality and performance
of the UFS driver. In some of the tests ufs_test uses several scsi
and UFS APIs for getting the disc information and sending UFS
specific commands.
The used APIs should be exposed in order to allow compilation
of ufs_test as a module.
Change-Id: I1263429bd3d5172af3b5552f0b8b503e32a04e51
Signed-off-by: Gilad Broner <gbroner@codeaurora.org>
Signed-off-by: Maya Erez <merez@codeaurora.org>
Adding a new access api for gendisk without using scsi private
headers.
This patch fixes multiple errors:
1. Relative include path in ufs_test.c to <../sd.h>
2. Allowing suspend after first test run.
Change-Id: I152d34667ab63790b643abb55111fc5b67c90ac9
Signed-off-by: Dolev Raviv <draviv@codeaurora.org>
[merez@codeaurora.org: fix trivial conflicts in ufs_test.c]
Signed-off-by: Maya Erez <merez@codeaurora.org>
[venkatg@codeaurora.org: resolved trivial merge conflicts,
use scsi_disk_get instead of __scsi_disk_get]
Signed-off-by: Venkat Gopalakrishnan <venkatg@codeaurora.org>
Add missing test name strings to long random tests.
These test names are used to distinguish between test case when running
automatic tests.
Change-Id: I1d375eb3cfdd2ae645ba086b7e3121026f3652ae
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
Currently, request sector is incremented in the long sequential test
using a calculation which assumes the device logical block size is
512B (equal to sector size) but this is not the case.
This patch changes the calculation to be correct, using the test bio size,
and assuming sector size is 512B.
Change-Id: I807be243b65834edf9201488a849777e43bb658a
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
Signed-off-by: Yaniv Gardi <ygardi@codeaurora.org>
Generalize the long_sequential_test framework to execute long random read
and long random write tests as well.
The random tests submit a large number of 4KB requests to randomly chosen
LBAs. Currently the tests are defined to read/write 64MB.
Change-Id: I9d797c0d61d863b2a9fbcd593a1be9065202ef76
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
[merez@codeaurora.org: fix trivial conflicts]
Signed-off-by: Maya Erez <merez@codeaurora.org>
This patch adds a completion condition to the long sequential unit test.
The condition is the completion of ALL the requests that where created.
Before this condition, the calculated test throughput was not accurate
since the transfered size was not constant.
This patch also increases the request size to the maximum - 128 bios.
In addition, the calculated throughput is always printed to the log.
Change-Id: Ibc16dc60486ff2b58cbc8ce02aee2ad87fea7ba1
Signed-off-by: Lee Susman <lsusman@codeaurora.org>
