Neural control on multiple time scales: Insights from human stick balancing

The time-delayed feedback control mechanisms of the nervous system are continuously subjected to the effects of uncontrolled random perturbations (herein referred to as noise). In this setting the statistical properties of the fluctuations in the controlled variable(s) can provide non-invasive insights into the nature of the underlying control mechanisms. We illustrate this concept through a study of stick balancing at the fingertip using high speed motion capture techniques. Experimental observations together with numerical studies of a stochastic delay differential equation demonstrate that on time scales short compared to the neural time delay (" fast control "), parametric noise provides a non-predictive mechanism that transiently stabilizes the upright position of the balanced stick. Moreover, numerical simulations of a delayed random walker with a repulsive origin indicate that even an unstable fixed point can be transiently stabilized by the interplay between noise and time delay. In contrast, on time scales comparable to the neural time delay (" slow control "), feedback and feedforward control mechanisms become more important. The relative contribution of the fast and slow control mechanisms to stick balancing is dynamic and, for example, depends on the context in which stick balancing is performed and the expertise of the balancer.


Introduction
It is becoming clear that biological systems tuned at or close to a stability boundary can exhibit qualitatively the same kinds of complex dynamics [1,2] that are observed for physical systems at or close to a critical point [3], including power laws, intermittency, and critical slowing down.A case in point arises in the neural control of stick balancing at the fingertip.In the control literature, the studies of the inverted planar pendulum figure prominently in discussions of the stabilization of an unstable fixed point [4].Since the equations of motion take the form [5] where is the length of the stick, g is the gravitational constant, γ is the damping coefficient, and θ is the vertical displacement angle (θ = 0 corresponds to the upright position, hence the "-" sign), the control problem is how to design the feedback, F , to stabilize an unstable saddle point.However, in real applications, such as those that arise in the context of neural control, it is necessary to take into account the time required to detect a deviation in θ, and then effect a corrective movement.Consequently the feedback is time-delayed and (1) becomes where τ is the time delay, and θ(t), θ(t − τ )) refer, respectively, to the values of θ at times t and t − τ .Since the upright position for (2) can be stable provided that τ and are appropriately chosen [6][7][8], the goal is to figure out the nature of the control strategy used by the nervous system from an analysis of the observed fluctuations.This neuro-biological inverse problem has attracted great interest [9,10].It is widely held that an understanding of how the human nervous system addresses the problem of stick balancing at the fingertip may provide useful insights into solving the problems ranging from minimizing the risk of falling in the elderly to the developing of stable two-legged robots [11].The observation that longer sticks are easier to balance at the fingertip than shorter ones is a reflection of the fact that the neural latency is finite (∼ 100 − 200 msec): as the stick becomes longer the speed of its movements eventually becomes slow enough so that the nervous system has time to make a corrective movement.However, it is found that no matter how long the stick or how expert the individual may be, the stick always eventually falls.Thus the intriguing question for stick balancing is not why the stick can be balanced at the fingertip, but to understand why the balanced stick eventually falls!
Here we review our studies on stick balancing at the fingertip [2,6,[12][13][14][15][16]. We demonstrate that it is possible to gain insight into the nature of the control strategy used by the nervous system by carefully studying the statistical properties of the fluctuations made in the controlled variable, i.e. θ, and the controller (i.e. the movements of the fingertip).

Survival functions
The simplest hypothesis to account for the observation that a stick balanced at the fingertip falls is that the dimensions of the basin of attraction associated with the stable upright position are small compared to the intensity of the random perturbations, herein referred to as noise.In this situation, for example, a sufficiently large perturbation occurring at the right time could move the trajectory outside the basin of attraction and hence produce a fall.If the hypothesis were correct, then the survival function, i.e. the fraction of trajectories remaining in the basin of attraction at time t, might be anticipated to be exponential.Indeed, exponential survival curves have been associated with the times to cross a threshold in chaotic dynamical systems [17,18] and in stochastic dynamical systems subjected to the effects of additive noise [19], i.e. the effects of noise are independent of the state variable.An example of a stochastic dynamical system with additive noise can be obtained by adding a noise term to the right-hand side of (2). Figure 1 (a) shows the survival curve, i.e. the probability that stick is balanced at the fingertip at time t, P (t esc > t), as a function of t, for two different stick lengths.A longer stick having the same mass is easier to balance and hence the survival curve is shifted to the right.As can be seen the survival curve is clearly not an exponential, but has the form of a Weibul function (figure 1(b)), where β > 1, λ are constants.Weibul-type survival functions have been observed previously in the context of the analysis of the failure times of ball bearings [20] and in survival statistics of patients with certain forms of cancer [17].
One mechanism that can generate survival functions having this form occurs when the effects of noise are parametric, i.e. the effects of noise enter through a parameter that is multiplied by the state variable [12].Vertical displacements in the pivot point of an inverted pendulum enter the equations of motion through a parameter [5]; hence parametric noise is particularly relevant to the study of stick balancing at the fingertip [21].If we expand F as a Taylor series and assume that the gain, r 0 varies noisily, then (2) becomes where and R 0 is a constant and ξ(t) is Gaussian distributed white noise with zero mean and variance σ.
The survival curves in figure 1 can be reproduced by discretizing (3) to obtain the second-order stochastic discrete dynamical system with state-dependent noise (compare triangles with the solid line for the right survival curve in figure 1) where and where r 0 is a constant, k/m = g/ , and ξ t is a random variable chosen from a Gaussian distribution of discrete variables having mean equal to zero and variance, σ.Thus, the observations in figure 1 point to a dynamical system in which parametric noise plays an important role in shaping the observed dynamics.Further insight into the mechanisms of the control of stick balancing can be obtained by examining the fluctuations made in the controlled variables.

Stick and fingertip motions
The fluctuations in the vertical displacement angle, θ, and the controller, i.e. the fingertip, can be measured with high accuracy using motion capture technology [6,12,13].Infrared light reflected from markers attached to each end of the stick are detected by two specialized motion capture cameras (Qualisys, Inc): the image projected onto the CCD of each camera determines two of the spatial coordinates and the third is determined using triangulation techniques involving both cameras.Measurements can be easily made with a temporal resolution of 1 msec and a spatial resolution of 4µm.(i.e. the marker closest to the fingertip) in the (x, y)-plane during stick balancing.These movements are also characterized by two components: 1) relatively slow translations of the pivot point for the inverted pendulum; and 2) small amplitude corrective movements.Taken together the observations in figure 2 suggest that it is convenient to consider the mechanisms for neural control on two time scales: 1) fast control, i.e. the time between successive corrective movements occur on time scales short compared to the neural time delay; and 2) slow control, i.e. the times between successive corrective movements occur on time scales that are comparable to the neural time delay.

Fast components for balance control
The times between successive crossings of a threshold (dashed horizontal line in figure 2 (a)) in the upward, or corrective, direction were measured.These times correspond to the laminar phases of a dynamical system that exhibits intermittency.Figure 3 (a) shows a double-log plot of the probability that the laminar phases have length δt, P (δt), as a function of δt.There are two important observations [6].First, > 98% of the laminar phases are shorter than the measured neural latency.Second, the fluctuations exhibit two statistical features of on-off intermittency [22], namely, the laminar phases exhibit a -3/2 power law (solid line in figure 3 (a) and the power spectrum of the fluctuations in θ exhibits a -1/2 power law (figure 3 (b)).
The power laws that characterize on-off intermittency can be reproduced using (3) provided that the parameters are chosen to place the deterministic version of this equation close enough to the stability boundary so that the r 0 (t) can be stochastically forced back and forth across it [6].These observations suggest that parametric noise plays an important role in the fast component of the control of stick balancing.It should be noted that the zero crossings for a random walk also scale as -3/2 [23].Thus the beneficial effects of on-off intermittency for stick balancing likely arise because the fluctuations in θ resemble a random walk for which the mean value of θ is approximately zero, i.e. the balance position is statistically stabilized [24].
In summary, the fast corrective movements for stick balancing exhibit three characteristics: 1) they are intermittent and ballistic; 2) the time between successive corrective movements is much shorter than the neural latency; and 3) they are open-loop, i.e. non-predictive (see below).It has been recently suggested that ballistic fluctuations that scale as -3/2 are associated with optimal control [25].Fast components having these same properties have also been observed in other human balancing tasks, namely postural sway during quiet standing [26,27] and ankle movement controlled balancing of an inverted pendulum [28].Although it has been suggested that the mechano-elastic properties of limbs and joints could provide a "zero delay" feedback control mechanism for balance [27,28], our observations on stick balancing suggest that the effects of parametric noise may be a more plausible explanation for these fast controlling movements.In addition, these observations suggest that methods used to characterize intermittent behavior in turbulent flow [29], such as the flatness factor, may be useful for the study of skill acquisition in humans.Data is from [6].

Stabilizing with noise and delay
An important question concerns whether the control system is tuned slightly to the stable or to the unstable side of the stability boundary, the proximity to the stability boundary depending on the intensity of the state-dependent noise.Two experimental observations suggested that the system may actually be tuned slightly to the unstable side of the boundary: 1) power laws identical to those observed for stick balancing are observed in a virtual stick balancing task that does not possess a stable point [14] and 2) the measured survival functions can be best fit by (4) when the parameters are chosen so that the deterministic version possesses only an unstable fixed point (see figure 1).The effect of partial or even full stabilization of unstable states by noise is well known to occur in stochastic dynamical parametric systems [30].Furthermore, recently it has been shown that additive noise can induce transient stability by delaying the escape time of unstable systems in periodically modulated systems [31] and in pure unstable damped systems [32].In the case of time delayed dynamical systems the stabilizing effect of additive and parametric noise by postponement of bifurcations has been also reported for specific systems [33].Thus it seems appropriate to ask ourselves whether there exist cooperative effects between random perturbations and time delay feedback at a more fundamental level responsible for improved stabilization in simple unstable systems.
To address this question we have considered the case of the repulsive delayed random walk [15] which, in turn, is a special case of the delayed random walk (DRW) [34].In the DRW the walker takes a discrete step of unit length per unit time in a direction determined by a set of conditional probabilities that depend on the position of the walker at a given time in the past, τ .The conditional probabilities for the walker to take a step to the right are where X(t) is the position of the walker at time t and 0 < p < 1.This walk differs from the well known brownian walk not only because its dynamics depend on the past but also because its origin can behave attractively or repulsively depending on the value of p.In particular the origin, X = 0, is attractive if p < 1/2, unbiased for p = 1/2 (as in the case of the brownian random walk) and repulsive if p > 1/2.Models of delayed random walks with an attractive origin enable a number of important properties of the delayed Langevin equation to be obtained analytically, including the autocorrelation function [34].However, analytical insights in the repulsive case are not immediate and thus we restrict our exposition to a numerical approach.Computer simulations of a DRW with a repulsive origin for a given τ require the definition of an initial function, Φ(s), where s ∈ [−τ, 0].We generated this function by using a brownian random walk with p = 1/2 and τ = 0.The origin of the delayed random walk corresponds to Φ(0).For t > 0 the position of the walker evolves while being influenced by its history at τ , X(t − τ ).To study the stability properties in the repulsive regime we concentrate our attention on the escape behavior.Thus the walker is said to escape at a first passage time L if at such time the walker crosses an arbitrarily established threshold ±X * .The question we ask is how long the walker can be kept around the repulsive origin without escaping?Figure 4 (a) depicts the mean first passage time (MFPT), L , for values of 1/2 < p 1. The MFPT shows a non-monotonous dependence on the time delay showing a maximum at a particular value of the time delay, τ * .When p increases from 1/2 this optimal delay time moves towards smaller values.
The behavior observed in the repulsive regime should be contrasted to that obtained for a DRW in the attractive regime, i.e. for p < 1/2.In this situation the MFPT also depends on the time delay.In particular, in the zero delay case the walker remains trapped around the origin for long times.As the time delay increases the MFPT rapidly and monotonously decreases indicating that the inclusion of delay in the dynamics destabilizes the walker.Moreover, the MFPT tends to decrease with increasing p and, as expected, it reaches a constant value when p = 1/2.
The mechanism by which this transient stabilization of the unstable origin occurs is still under investigation.Preliminary observations suggest that it reflects the interplay between two opposing tendencies: 1) the stabilizing effects of the past history of the walker that delay escape, and 2) the greater the probability of escape the longer the walk is.

Slow control
The above mechanism for fast control does not readily explain why, for example, a stick cannot be balanced at the fingertip so well with the eyes closed as with the eyes open.Obviously closedloop feedback control mechanisms that depend, in part, on visual and other sensory inputs should also be involved in this task.We refer to these unidentified controlling mechanisms as the "slow components" for control since they cannot be expected to respond quicker than the related neural conduction times.The identification of the nature of these slow control mechanisms has been more difficult [10].The observation that closed-loop feedback control of voluntary movements become of lesser importance as expertise develops [35], suggests that it might be possible to gain insights into their role by identifying the controlling movements that occur as expertise develops.Comparison of the probability distribution of ∆V of hand movements during stick balancing for the same person at two different skill levels [13]: low skill (•) and high skill ( ).The stick length was 62 cm.The reflector closest to the fingertip was used to track the hand movements.The vertical dashed lines show the truncation point: low skill at 0.26 and high skill at 0.41.Similar changes in the probability distributions with the development of stick balancing expertise have been obtained in four individuals (in preparation).Data is taken from [13].
Suppose we model the movements of the fingertip during stick balancing as a random walk.The step size of the random walker per unit time is equal to the speed, V .In order to focus on the fast components of these movements (figure 2 (b)) it is necessary to high pass filter the time series.This is equivalent to calculating the change in speed, ∆V [13].Figure 5 compares the distribution in the changes of the distribution of ∆V , P (∆V ),that occur when a subject was a novice stick balancer and that obtained when they became more expert after 8 days of intensive practice.The increase in expertise coincides with a change in the shape of P (∆V ), i.e. the distribution develops fatter tails at the higher skill level.Surprisingly the Lévy exponent was found to be ∼ 0.9 [13].Lévy flights with α ≈ 1 have the characteristics of super-diffusion and have been shown to be optimal for a variety of random search patterns [36].This observation suggests that the fast components of the control of stick balancing resemble a random foraging strategy.
A simple explanation for the changes shown in figure 5 is that they reflect differences in trun-cation [37,38].If we write where L α (∆V ) describes the Lévy distribution in ∆V ) with Lévy index α, and c 1 , c 2 are normalization constants.The truncation function, f (∆V ), is typically a decreasing function of ∆V with compact support and the truncation threshold, Π, is the critical value of ∆V at which the distribution begins to deviate from L α (∆V ).Truncation accounts for the fact that ∆V is limited by both the biomechanical properties of the musculoskeletal system (which are unlikely to change over the training period of 1-2 weeks) and the effects of time-delayed (reactive) feedback mechanisms: the more active the feedback control, the more truncated [38].No skilled-related changes in the Lévy index, α, or the Lévy scale factor were observed [13].Thus the slow components for the control of stick balancing that change as the expertise develops, are likely to either involve a reactive feedback at the level of local sensorimotor loops (e.g.[7]) or possibly the mechanisms based on feedforward control involving higher neural centers (e.g.[10]).

Discussion
The parametric noise mechanism we have described for the fast component of the control for stick balancing will obviously be most important when the movements of the stick are quick compared to the neural time delay.Although this observation would argue that this mechanism should be of lesser importance once the stick movements become sufficiently sluggish, e.g.long and heavier sticks, experimental observations indicate that the situation is far from being that simple.For example, novice stick balancers are capable of improving performance, i.e. shift the stick survival curve to the right, by concurrently moving a leg rhythmically [15].The fact that a similar improvement in performance can be obtained by having the subject think about moving the leg rhythmically, but not actually moving it [16], demonstrates that this improvement is not simply due to mechanical vibrations of the fingertip induced by leg movement.The simplest explanation is that the increase in performance is the result of the diversion of the intention for voluntary corrective movements away from the stick balancing task.In other words the improvement in performance arises because a corrective movement guided by volition is not necessarily optimal and hence their role should be minimized since they may do more harm than good.
At any instant in time the control of a voluntary movement by the nervous system reflects the interplay between multiple controlling effects.Closed-loop neural feedback loops in which information obtained from sensory receptors to adjust corrective movements are organized in a hierarchical manner [39]: local reflex control loops are organized into cooperative synergies under the control of neural centers located at progressively higher neural centers (e.g.spinal cord, brainstem, cortex).As the environment and the demands of the task become more predictable, it becomes possible to use feed-forward mechanisms to program the neural activations before the movements should be made.Moreover, it follows directly from the equations of motion for every motor task, that control cannot be completely due to the effects of neural activations alone, but should also take into account the biomechanical properties of the musculoskeletal system and the changing nature of the environment in which the movements occur [40].Often neglected is the fact that the careful identification of noise sources and their effects on variability can provide us with important non-invasive methods to gain insights into the structure of the underlying control networks [2,41].
What are the rules that govern the interplay between all of these controlling effects?To what extent are these rules dependent on factors such as the nature of the task, the expertise of the performer, and the demands placed on critical, but limited resources, e.g.attention?Stick balancing at the fingertip provides an important and accessible experimental paradigm to investigate these issues using non-invasive techniques.The development of virtual stick balancing tasks that involve the interplay between a human and a computer [10,14] will enable these studies to be extended into the functional neuroimaging suite and possibly to even the study of non-human primates (e.g.[42]).
Thus obtaining an understanding of how the nervous system controls complex voluntary tasks may be no further away than the tip of our finger!

Figure 2 (Figure 2 .
Figure 2. (a) Temporal series for ∆z/l for a 62 cm stick balanced at the fingertip.The horizontal dashed line depicts the threshold position.(b) Two-dimensional representation of the path traveled by the fingertip during stick balancing.The digitization timestep was 1 ms.Movements in the anterior-posterior direction are shown on the y axis; side to side movements on the x axis.Data is taken from[13].

Figure 3 .
Figure 3. (a) Log-log plot of the normalized probability of having laminar phases of length δt, P (δt) for a balanced stick of length 62 cm.The dotted line has slope −3/2 and the physiological delay is depicted by the vertical line.(b) Power spectrum, S(f ), of the fluctuations in θ showing the presence of two power laws.The slopes of the solid lines are, respectively, −0.5 and −2.5.Data is from[6].

Figure 4 .
Figure 4. Mean first passage time as a function of the delay time for X * = 120 steps and different values of the bias p.(a) From top to bottom: p = 0.7, 0.8, 0.9, 1.0.(b) From top to bottom: p = 0.425, 0.45, 0.475, 1/2.Data obtained with 100000 realizations.

Figure 5 .
Figure 5.Comparison of the probability distribution of ∆V of hand movements during stick balancing for the same person at two different skill levels[13]: low skill (•) and high skill ( ).The stick length was 62 cm.The reflector closest to the fingertip was used to track the hand movements.The vertical dashed lines show the truncation point: low skill at 0.26 and high skill at 0.41.Similar changes in the probability distributions with the development of stick balancing expertise have been obtained in four individuals (in preparation).Data is taken from[13].