There is no real need to control HMP and power-aware task placement at
runtime after kernel has booted. Boot-time control should be
sufficient. Not allowing for runtime (sysctl) support simplifies the
code quite a bit.
Also rename sysctl_sched_enable_hmp_task_placement to be shorter.
Change-Id: I60cae51a173c6f73b79cbf90c50ddd41a27604aa
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed minor conflict. p->nr_cpus_allowed == 1
has moved to core.c
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
It should be possible to bypass all HMP scheduler changes at runtime
by setting sysctl_sched_enable_hmp_task_placement and
sysctl_sched_enable_power_aware to 0. Fix various code paths to honor
this requirement.
Change-Id: I74254e68582b3f9f1b84661baf7dae14f981c025
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed conflict in rt.c, p->nr_cpus_allowed ==
1 is now moved in core.c]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
Avoid the long if() block of code in set_task_cpu(). Move that code to
its own function
Change-Id: Ia80a99867ff9c23a614635e366777759abaccee4
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
It would be fatal if task_cpu() information for a task does not
accurately represent the cpu on which its running. All sorts of wierd
issues can arise if that were to happen! Add a BUG_ON() in context
switch to detect such cases.
Change-Id: I4eb2c96c850e2247e22f773bbb6eedb8ccafa49c
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Avoid wasting effort in migrating tasks that are about to sleep.
Change-Id: Icf9520b1c8fa48d3e071cb9fa1c5526b3b36ff16
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
Fix the hack to set task's on_rq to 0 during task migration. Task's
load is temporarily added back to its runqueue so that
update_task_ravg() can fixup task's load when its demand is changing.
Task's load is removed immediately afterwards.
Temporarily setting p->on_rq to 0 introduces a race condition with
try_to_wake_up(). Another task (task A) may be attempting to wake
up the migrating task (task B). As long as task A sees task B's
p->on_rq as 1, the wake up will not continue. Changing p->on_rq to
0, then back to 1, allows task A to continue "waking" task B, at
which point we have both try_to_wake_up and the migration code
attempting to set the cpu of task B at the same time.
CRs-Fixed: 695071
Change-Id: I525745f144da4ffeba1d539890b4d46720ec3ef1
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Currently active_load_balance_cpu_stop is run by cpu stopper and it
pushes running tasks off the busiest CPU onto idle target CPU. But
there is no check to see whether target cpu is offline or not before
pushing the tasks. With the introduction of active migration in the
scheduler tick path (see check_for_migration()) there have been
instances of attempts to migrate tasks to offline CPUs.
Add a check as to whether the target cpu is online or not to prevent
scheduling on offline CPUs.
Change-Id: Ib8ac7f8aeabd3ca7365f3eae977075952dab4f21
Signed-off-by: Prasad Sodagudi <psodagud@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
In the absence of a power driver providing real power values, the scheduler
currently defaults to using capacity of a CPU as a measure of power. This,
however, is not a good measure since the capacity of a CPU can change due
to thermal conditions and/or other hardware restrictions. These frequency
restrictions have no effect on the power efficiency of those CPUs.
Introduce max possible capacity of a CPU to track an absolute measure of
capacity which translates into a good absolute measure of power efficiency.
Max possible capacity takes the max possible frequency of CPUs into account
instead of max frequency.
Change-Id: Ia970b853e43a90eb8cc6fd990b5c47fca7e50db8
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Interrupts can end up waking processes on the same cpu as the one for
which sched_get_busy() is called. Since sched_get_busy() takes the rq
lock this can result in a deadlock as the same rq lock is required to
enqueue the waking up task. Fix the deadlock by disabling interrupts
when taking the rq lock.
Change-Id: I46e14a14789c2fb0ead42363fbaaa0a303a5818f
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
update_task_ravg() in context switch uses wallclock that is updated
before running put_prev_task() and pick_next_task(), both of which can
take some time. Its better to update wallclock after those routines,
leading to more accurate accounting.
Change-Id: I882b1f0e8eddd2cc17d42ca2ab8f7a2841b8d89a
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Load calculations have been modified to accept and return 64 bit values.
Fix up all the places where we make such calculations to store the result
in 64 bit variables. This is necessary to avoid issues caused by
truncation of values.
While at it update scale_task_load() to scale_load_to_cpu(). This is
because the API is used to scale load of both individual tasks as well as
the cumulative load of CPUs. In this sense the name was a misnomer. Also
clean up power_cost() to use max_task_load().
Change-Id: I51e683e1592a5ea3c4e4b2b06d7a7339a49cce9c
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed conflict in power_cost(). power_cost()
now supports sched_use_pelt=1 back again.
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
Small tasks execute for small durations. This means that the power
cost of taking CPUs out of a low power mode outweigh any performance
advantage of using an idle core or power advantage of using the most
power efficient CPU. Introduce C-state aware task placement for small
tasks. This requires a two pass approach where we first determine the
most power effecient CPU and establish a band of CPUs offering a
similar power cost for the task. The order of preference then is as
follows:
1) Any mostly idle CPU in active C-state in the same power band.
2) A CPU with the shallowest C-state in the same power band.
3) A CPU with the least load in the same power band.
4) Lowest power CPU in a higher power band.
The patch also modifies the definition of a small task. Small tasks
are now determined relative to minimum capacity CPUs in the system
and not the task CPU.
Change-Id: Ia09840a5972881cad7ba7bea8fe34c45f909725e
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Certain low latency bursty workloads require immediate use of highest
capacity CPUs in HMP systems. Existing load tracking mechanisms may be
unable to respond to the sudden surge in the system load within the
latency requirements. Introduce the scheduler boost feature for such
workloads. While boost is in effect the scheduler bypasses regular load
based task placement and prefers highest capacity CPUs in the system
for all non-small fair sched class tasks. Provide both a kernel and
userspace API for software that may have apriori knowledge about the
system workload.
Change-Id: I783f585d1f8c97219e629d9c54f712318821922f
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed minor conflict in
include/linux/sched/sysctl.h.]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
select_best_cpu() invokes trace_sched_cpu_load() for all online cpus
in a loop, before it enters the loop for core selection. Moving
invocation of trace_sched_cpu_load() in inner core loop is potentially
more efficient.
Change-Id: Iae1c58b26632edf9ec5f5da905c31356eb95c925
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
balance_interval needs to be reset for anycpu being kicked. Otherwise it
can end up ignoring the kick (i.e not doing load balance for itself).
Also bypass the check for existence of idle cpus in tickless state for
!CONFIG_SCHED_HMP to allow for more aggressive load balance.
Change-Id: I52365ee7c2997ec09bd93c4e9ae0293a954e39a8
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed minor conflict in nohz_kick_needed().]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
We currently check the need to migrate the currently running task in
scheduler_tick(). Skip that check for small tasks, as its not worth
the effort!
Change-Id: Ic205cc6452f42fde6be6b85c3bf06a8542a73eba
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
In estimating cost of running a task on a given cpu, cost of cpu at
its current frequency needs to over-ride cost at frequency demanded by
task, where cur_freq exceeds required frequency of task. This is
because placing a task on a cpu can only result in an increase of
cpu's frequency.
Change-Id: I021a3bbaf179bf1ec2c7f4556870256936797eb9
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Window-based load tracking is a pre-requisite for the scheduler to
feed cpu load information to the governor. When PELT is in use, return
failure when governor attempts to set window-size. This will let
governor fall back to other APIs for retrieving cpu load statistics.
Change-Id: I0e11188594c1a54b3b7ff55447d30bfed1a01115
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed trivial merge conflict
in include/linux/sched.h.]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
Provide information in /proc/sched_debug on min_capacity, max_capacity
and whether pelt or window-based task load statistics is in use.
Change-Id: Ie4e9450652f4c83110dda75be3ead8aa5bb355c3
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Move up chunk of code to be defined early. This helps a subsequent
patch that needs update_min_max_capacity()
Change-Id: I9403c7b4dcc74ba4ef1034327241c81df97b01ea
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed trivial conflicts.]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
During bootup, its possible for min_max_freq to change as frequency
information for additional clusters is processed. That would need to
trigger recalculation of capacity/load_scale_factor for all (online)
cpus, as they strongly depend on min_max_freq variable. Not doing so
would imply some cpus will have their capacity/load_scale_factor
computed wrongly.
Change-Id: Iea5a0a517a2d71be24c2c71cdd805c0733ce37f8
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
A CPU may have its frequency changed by a different CPU. Because of
this, it is not guaranteed that we will update task statistics at
approximately the same time that the frequency change occurs. To
guard against accruing time to a task at the wrong frequency, update
the task's window-based statistics if the CPU it is running on
changes frequency.
Change-Id: I333c3f8aa82676bd2831797b55fd7af9c4225555
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Add trace events for update_task_ravg(), update_history(), and
set_task_cpu(). These tracepoints are useful for monitoring the
per-task and per-runqueue demand statistics.
Change-Id: Ibec9f945074ff31d1fc1a76ae37c40c8fea8cda9
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Rebalancing at exec time will currently undo any beneficial placement
that has been done during fork time, since select_best_cpu() will not
discount the currently running task.
For now just skip re-evaluating task placement at exec.
Change-Id: I1e5e0fcc329b7b53c338c8c73795ebd5e85a118b
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Historical load maintained per task can be used to influence cpu
frequency better. For example, when a heavy demand task wakes up after
prolonged sleep, we could use the historical load information to alert
cpufreq governor about the need to raise cpu frequency. This patch
changes CPU busy statistics to be aggregation of historical task
demand. Also task's historical load (as defined by
sysctl_sched_window_stats_policy) is add to cpu's busy statistics
(rq->curr_runnable_sum) whenever it executes on a cpu.
Change-Id: I2b66136f138b147ba19083b9b044c4feb20d9b57
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
In the event that a full window (or multiple full windows) have
elapsed when updating a task's window-based stats, the runtime of
those windows needs to be scaled based on the CPU frequency. This
is currently missing, causing full windows to be accounted as having
elapsed at maximum frequency, erroneously inflating task demand.
Change-Id: I356b4279d44d4f39c8aea881c04327b70ed66183
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
After a migration occurs the source and destination CPUs may
not be running at frequencies which match the new task load on
those CPUs.
Previously, the scheduler was notifying cpufreq anytime a task
greater than a certain size migrates. This is suboptimal however
since this does not take into account the CPU's current
frequency and other task activity that may be present.
Change-Id: I5092bda3a517e1343f97e5a455957c25ee19b549
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
When the number of tasks intended for a cluster exceed the number of
mostly idle CPUs in that cluster, the scheduler currently freely uses
CPUs in other clusters if possible. While this is optimal for
performance the power trade off can be quite significant. Introduce
spill threshold tunables that govern the extent to which the scheduler
should attempt to contain tasks within a cluster.
Change-Id: I797e6c6b2aa0c3a376dad93758abe1d587663624
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
[joonwoop@codeaurora.org: fixed conflict in nohz_kick_needed()]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
Knowing the affinity mask and CPU usage of a task is helpful
in understanding the behavior of the system. Affinity information
has been added to the enq_deq trace event, and the migration
tracepoint now reports the load of the task migrated.
Change-Id: I29d8a610292b4dfeeb8fe16174e9d4dc196649b7
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
When a task moves between CPUs in two different frequency domains
the cpufreq governor may wish to immediately modify the frequency
of both the source and destination CPUs of the migrating task.
A tunable is provided to establish what size task is considered
"significant" enough to warrant notifying cpufreq.
Also fix a bug that would cause load to not be accounted properly
during wakeup migrations.
Change-Id: Ie8f6b1cc4d43a602840dac18590b42a81327c95a
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
[rameezmustafa@codeaurora.org: Add double rq locking for set_task_cpu()]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Should the system get stuck in a state where load balancing is failing
due to all tasks being pinned, deferring load balancing for up to half
a second may cause further performance problems. Eventually all tasks
will not be pinned and load balancing should not be deferred for a great
length of time.
Change-Id: I06f93b5448353b5871645b9274ce4419dc9fae0f
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
If we have a task to schedule, we currently don't consider CPUs where
it will not fit even if they are idle. Instead we choose the previous
CPU which is sub-optimal for performance if an idle CPU is
present. This change introduces tracking of any idle CPUs irrespective
of whether the task fits on them or not. If we don't have a good place
to put the task, prefer the lowest power idle CPU.
Change-Id: I4e8290639ad1602541a44a80ba4b2804068cac0f
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
Real Time task scheduling has historically been geared towards performance
with a significant attempt to keep higher priority tasks on the same CPU.
This is not optimal for power since the task CPU may not be the most
power efficient CPU.
Also task movement via select_lowest_rq() gives CPU priority the primary
consideration before looking at CPU topologies to find a CPU closest to
the task CPU in terms of topology. This again is not optimal for power
since the closest CPU may be significantly worse for power than CPUs
further away.
This patch removes any bias for the task CPU. When the lowest priority
CPUs in the system are found we give no consideration to the CPU topology.
Instead we find the lowest power CPU within local_cpu_mask. This takes care
of select_task_rq_rt() and push_task(). The pull model remains unaffected
since we have no room for power optimization there.
Change-Id: I4162ebe2f74be14240e62476f231f9e4a18bd9e8
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[joonwoop@codeaurora.org: s/__get_cpu_var/this_cpu_ptr/]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
Normally the load balancer does not pay attention to CPUs with one
task since it is not possible to subdivide that load any further
to achieve better balance. With power aware scheduling however it
may be desirable to relocate that one task if the CPU it is currently
executing on is less power efficient than other CPUs in the system.
Change-Id: Idf3f5e22b88048184323513f0052827b884526b6
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
[joonwoop@codeaurora.org: group_capacity_factor changed to
group_no_capacity. group_classify is called by update_sg_lb_stats() too.]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
Although tasks are routed to the most power-efficient CPUs during
task wakeup, a CPU-bound task will not go through this decision point.
Load balancing can help if it is modified to dislodge a single task
from an inefficient CPU. The situation can be further improved if
during the tick, the task placement is checked to see if it is
optimal.
This sort of checking is already being done to ensure proper
task placement in heterogneous CPU topologies, so checking for
power efficient task placement fits pretty well.
Change-Id: I71e56d406d314702bc26dee1438c0eeda7699027
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
The nohz load balancer CPU does load balancing on behalf of all
idle tickless CPUs. In the interest of power efficiency though, we
should do load balancing on the most power efficient idle tickless
CPU first, and then work our way towards the least power efficient
idle tickless CPU. This will help load find its way to the most
power efficient CPUs in the system.
Since when selecting the CPU to balance next it is unknown what
task load would be pulled, a frequency must be assumed in order
to do a comparison of CPU power consumption. The maximum
freqeuncy supported by all CPUs is used for this.
Change-Id: I96c7f4300fde2c677c068dc10fc0e57f763eb9b2
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[joonwoop@codeaurora.org: fixed minor conflict in nohz_idle_balance().]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
When a CPU goes idle, it checks to see whether it can pull any load
from other busy CPUs. The CPU going idle may not be the most
power-efficient idle CPU in the system however.
This patch causes the CPU going idle to check to see whether
there is a more power-efficient idle CPU within the same
lowest sched domain. If there is, then it runs the load balancer
on behalf of that CPU instead of itself.
Since it is unknown at this point what task load would be pulled,
a frequency must be assumed for this in order to do a comparison
of CPU power consumption. The maximum freqeuncy supported by all
CPUs is used for this.
Change-Id: I5eedddc1f7d10df58ecd358f37dba563eeecf4fc
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
[joonwoop@codeaurora.org: fixed minor conflict around comment.]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
To enable power-aware scheduling, provide a hook/infrastructure
for platforms to communicate CPU power requirements for each
supported CPU frequency. This information is then used to estimate
the cost of running a task on a given CPU.
Currently, an assumption is made that the task will be running
by itself on the CPU. Given the current policy tries to spread
tasks as much as possible this assumption should not be too
far off.
Change-Id: I19f1fa760a0d43222d2880f8aec0508c468b39bb
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[joonwoop@codeaurora.org: return rq->capacity as power cost with
sched_use_pelt=1. se.avg.runnable_avg_period is deprecated and
power_cost() will be changed by subsequent change anyway./
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
The sched_enable_power_aware sysctl will control whether
or not scheduling decisions are influenced by the power
consumption of individual CPUs.
Change-Id: I312f892cf76a3fccc4ecc8aa6703908b205267f0
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
This will make it easier to account interrupt time on a cpu,
introduced in a subsequent patch.
Change-Id: I0e1fb5255c280ca374fd255e7fc19d5de9f8b045
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org
sched_get_busy() returns the busy time of a cpu during the most
recent completed window.
sched_set_window() will set window size and aligns windows across
all CPUs.
Change-Id: Ic53e27f43fd4600109b7b6db979e1c52c7aca103
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed minor conflict in include/linux/sched.h]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
Adjust cpu's busy time in its recent and previous window upon task
migration. This would enable scheduler to provide better inputs to
cpufreq governor on a cpu's busy time in a given window.
Change-Id: Idec2ca459382e9f46d882da3af53148412d631c6
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Add counters per-cpu to track its busy time in the latest window and
one previous to that. This would be needed to track accurate busy time
per-cpu that accounts for migrations. Basically once a task migrates,
its execution time in current window is migrated as well to new cpu.
The idle task's runtime is not accounted since it should not count
towards runqueue busy time.
Change-Id: I4014dd686f95dbbfaa4274269bc36ed716573421
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Currently windows where tasks had no execution time are ignored.
However accurate accounting of cpu busy time that factors in migration
would need to know actual utilization of a task in the window previous
to the latest one. This would help scheduler guide cpufreq governor on
busy time per-cpu that is not subject to migration induced errors.
Change-Id: I5841b1732c83e83d69002139de3bdb93333ce347
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Synchronizing windows across cpus for task load measurements
simplifies cpu busy time accounting during migrations. For task
migrations, its usage in current window can be carried over to its new
cpu. This lets cpufreq governor see a correct picture of cpu busy time
that is not affected by migrations.
This patch lines up windows across cpus. One of the cpu, sync_cpu,
serves as a reference for all others. During bootup sync_cpu would
initialize its window_start (from its sched_clock()). Other cpus will
synchronize their window_start in reference to sync_cpu. This patch
assumes synchronous sched_clock() across cpus and may need some change
to address architectures which do not provide such synchronized
sched_clock().
Change-Id: I13381389a72f5f9f85cc2446401d493a55c78ab7
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Task load statistics are used for two purposes : cpu frequency
management and placement. Task's load can't be accurately judged by
its wait time. For ex: a task could have waited for 10ms and when given
opportunity to run, could just execute for 1ms. Accounting for 11ms as
task's demand could be over-stating its needs in this example. For
now, remove wait time from task demand and instead let task load be
derived from its actual exec time. This may need to become a tunable
feature.
Change-Id: I47e94c444c6b44c3b0810347287d50f1ee685038
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
During migrations accounting needs to be done in set_task_cpu() to
subtract the task activity from the source CPU and add it to the
destination CPU. This accounting will require that the task's window
based load statistics be up to date.
Unfortunately, the window-based statistics cannot always be updated in
set_task_cpu() because they are already being updated in the wakeup
path. We cannot update the statistics solely in the wakeup path
because not all wakeups are migrations. Those non-migrating wakeups
will not enter set_task_cpu().
To ensure the window-based stats are always updated for both wakeup
migrations and regular migrations, they are updated earlier in the
wakeup path, and also updated in set_task_cpu if the task is already
runnable (this ensures it is not a wakeup migration, but a regular
migration).
Change-Id: Ib246028741d0be9bb38ce93679d6e6ba25b10756
Signed-off-by: Steve Muckle <smuckle@codeaurora.org>
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
[joonwoop@codeaurora.org: fixed minor conflict in fair.c]
Signed-off-by: Joonwoo Park <joonwoop@codeaurora.org>
set_task_cpu() will need to call update_task_ravg(). Move up
definition to make it easy.
Change-Id: I95c1c9e009bd1805f28708e8d6fd3b7b2166410e
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
[rameezmustafa@codeaurora.org]: Port to msm-3.18]
Signed-off-by: Syed Rameez Mustafa <rameezmustafa@codeaurora.org>
Set window-based load stats to be the default mechanism under which
tasks get classified (as big/small) and which will drive frequency
demand for tasks. sched_ravg_window kernel parameter can be used to
change this default setting to use PELT (per-entity load tracking)
scheme instead.
Change-Id: I626110daa0bb2b53172bedea829d31877255ceaa
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
Provide a runtime tunable to switch between using PELT-based load
stats and window-based load stats. This will be needed for runtime
analysis of the two load tracking schemes.
Change-Id: I018f6a90b49844bf2c4e5666912621d87acc7217
Signed-off-by: Srivatsa Vaddagiri <vatsa@codeaurora.org>
