In this paper we are concerned with the so-called “supervised learning” or “learning from examples” problem, which refer to the task of predicting an output, say v ∈ V ⊂ ℝ, of a system under study from a previously unseen input u ∈ U ⊂ ℝ^d on the basis of the set of training examples, that is a set of input-output pairs z_i = (u_i, v_i), i = 1, 2, …, m, observed in the same system.

A kernel based algorithm consists in learning a function x : U → ℝ in some Reproducing Kernel Hilbert Space (RKHS) X = X_K generated by a suitable Mercer (continuous, symmetric and positive semidefinite) kernel K : U × U → ℝ that assigns to each input u ∈ U an output x(u). The prediction error is measured by the value of a chosen loss function l(v, x(u)), say square-loss l(v, x(u)) = (v − x(u))².

In the context of kernel based supervised learning a regularization is a compromise between the attempt to fit given data z={zi=(ui,vi)}i=1m⊂U×V and the desire to reduce the complexity of a data fitter x ∈ X_K. For example, a Tikhonov-type kernel based learning algorithm in its rather general form defines a predictor x(u) = x_z(u) = x(K; u) ∈ X_K as the minimizer of the sum:

where the complexity is measured by the norm ||·||_XK in RKHS generated by a kernel K, and the above-mentioned compromise is governed by the value of the regularization parameter α.

As it has been mentioned in Michelli and Pontil [1], a challenging and central problem of kernel based learning is the choice of the kernel K itself. Recent applications (see [2, 3]) have shown that using multiple kernels instead of a single one can enhance the interpretability of the predictor x(u) = x(K; u) and improve performances. In such cases, a convenient approach is to consider that the kernel K is actually a combination of predefined basis kernels:

where each basis kernel K_j may either use the full set of variables (u¹, u², …, u^d) describing the input u = (u¹, u², …, u^d) ∈ ℝ^d or subsets of variables stemming from different data sources. Defining both the minimizer x(K; u) of T_{α, z}(x; K) and the weights Equation (2), d_j, j = 1, 2, …, p, in a single optimization problem is known as the Multiple Kernel Learning (MKL).

A generalized representer theorem by Schölkopf et al. [4] tells us that a large class of optimization problems Equation (1) with RKHS have solutions expressed as kernel expansions in terms of the training data z. More precisely, for each K = K_j, j = 1, 2, …, p, the minimizer x(K_j; u) belongs to the finite-dimensional linear subspace spanned by the functions K_j(u, u_i), i = 1, 2, …, m. Therefore, MKL-predictor x_z(u) can be written as:

There is a significant amount of work in the MKL-literature. For example, the survey [5] reviews a total of 96 references. A comparison of existing MKL-algorithms performed in that survey shows that through there may not be large differences in terms of accuracy, there is difference between MKL-algorithms in complexity as given by the training time, for example.

Similar accuracies of different MKL-algorithms can be, at least partially, explained by the fact that all minimizers x_j(K_j; ·) of Equation (1) for K = K_j, j = 1, 2, …, p belong to the direct sum X_K+ of RKHS X_Kj, j = 1, 2, …, p, and a classical result [6] says that X_K+ is a RKHS generated by the kernel K₊ = K₁ + K₂ + … +K_p.

The differences in training time originate from different optimization algorithms used for learning the weights d_j, j = 1, 2, …, p. The so-called one-step methods calculate both the combination weights d_j, j = 1, 2, …, p and the parameters of the minimizer x(K; u) in a single run. However, these methods use optimization approaches, such as semi-definite programming (SDP) [7], second order conic programming (SOCP) [8], or quadratically constrained quadratic programming (QCQP) [9], which have high computational complexity especially for large number of samples.

Two-step, or wrapper methods address the MKL-problem by iteratively solving a single learning problem, such as Equation (1), for a fixed combination of basis kernels, and then update the kernel weights. Such methods run usually much faster than the one-step methods and therefore there is a rich variety of them. Just to mention some of them we refer to Semi-Infinite Linear Program (SILP) approach [10], SimpleMKL method [11] performing a reduced gradient descent on the kernel weights, LevelMKL algorithm [12], HessianMKL [13], replacing the gradient descent in SimpleMKL by the Newton update, and some others [1, 14].

The time complexity of these two-step methods depends on the number of iterations of an algorithm until the stopping criterion is fulfilled. In this context our idea is to use the so-called Linear Functional Strategy (LFS) [15], originally proposed for single-kernel ranking learning, allowing to find the coefficients of the linear combination in one step by using the given training set. In this paper we extend this idea to the MKL setting, namely we analyze the linear functional strategy for combining the x_j(K_j; u) ∈ X_Kj into a new predictor.

The paper is organized as follows: in the next section we present the main theoretical results. Then, we compare the proposed method with two other MKL-learning approaches, namely SimpleMKL and SpicyMKL [16]. Finally, we discuss some open questions and further research directions.

If we accept the basic statistical learning assumptions, namely that the inputs u and the outputs v are assumed to be related by a conditional probability distribution ρ(v|u) of v given u, and the input u is also assumed to be random and governed by an unknown marginal probability ρ_U on U so that there is an unknown probability distribution ρ(u, v) = ρ_U(u)ρ(v|u) on the sample space U × V from which the data forming the training set are drawn independently, then the “ideal predictor” x^† is assumed to belong to the space X = L₂(U, ρ_U) of square integrable functions with respect to ρ_U and can be defined as the element minimizing the expected prediction risk:

Moreover, in such set-up x^† can be written explicitly in the form:

As one can see from the above formula, x^† cannot be used in practice, since the conditional probability ρ(v|u) is not assumed to be known.

Our idea is to construct a MKL-predictor Equation (3) in the form:

where c_j are the unknown coefficients. It is clear that the best choice of the coefficient vector c=c¯ should minimize the distance:

and it is easy to see that such c¯ solves the following linear system of equations:

with the Gram matrix G=(〈xl(Kl;·),xj(Kj;·)〉L2(U,ρU))l,j=1p and the right-hand-side vector g=(〈x†,xl(Kl;·)〉L2(U,ρU))l=1p. Clearly, the right-hand side g is inaccessible due to the involvement of the unknown predictor x^†. Furthermore, the components of the matrix G are not accessible, since the measure ρ_U is also unknown.

To simplify further discussion, we assume that x†∈XK+, which is not a big restriction if at least one of K_j is the so-called universal kernel [17], and consider the inclusion operator I₊ of X_K+ into L₂(U, ρ_U) and the sampling operator Sz:XK+→ℝm defined by Szx=(x(u1),x(u2),…,x(um))∈ℝm. Then under rather general assumptions (see 18) with high probability 1 − η we have:

where Sz*:ℝm→XK+, I+*:L2(U,ρU)→XK+ are adjoints of S_z and I₊; v = (v₁, v₂, …, v_m) is the vector of the outputs used for training; c₁, c₂ are certain multipliers which do not depend on m, η.

With the use of Equation (7) we can approximate the components of the vector g and the Gram matrix G as:

In view of these relations, we state the main result of this section through the following theorem:

Theorem 1. Let V⊂[-b,b], x†∈XK+ and K_j(u, u) ≤ c, u ∈ U, j = 1, 2, …, p. Consider

and assume that G~ is invertible. Then for MKL-approximant:

with confidence 1 − η it holds:

where a coefficient implicit in O-symbol does not depend on m.

Proof. Formulas (8, 9) tells us that with confidence 1 − η it holds:

If G~-1 exists then in view of (11) it is natural to assume that for sufficiently large m with confidence 1 − η we have:

This assumption allows the application of the well-known Banach theorem on inverse operators (see 19, V. 4.5), which tells that:

Consider the vectors c¯=G-1g,c~=G~-1g~. Then from (11) to (13) it follows that:

and

Moreover, since x_j(K_j; ·) is the minimizer of Equation (1) for K = K_j and l(v, x(u)) = (v − x(u))², from Proposition 4.1 [20] it follows that ||x_j(K_j; ·)||_{L2(U, ρU)} are uniformly bounded such that for α from the range of interest, i.e., for α ≥ m⁻¹, we have ||xj(Kj;·)||L2(U,ρU)≤d||x†||L2(U,ρU), where d=O(log1η) and does not depend on m, j.

Then the statement of the theorem follows from (14) to (15).    □

Remark 1. In Theorem 1 the Gram matrix G~ is supposed to be well-conditioned. In principle, one may control this by excluding those members of the family {x_j(K_j; u), j = 1, 2, …, p} that are close to be linear dependent of others. A preconditioning method for such a control has been discussed, for example, in Chen et al. [21] (see Remark 3.1 there).

Theorem 1 tells us that the effectively constructed linear combination of the candidates x_j(K_j; ·), j = 1, 2, …, p, is almost as accurate as the best linear aggregator of them.

A simple algorithmic sketch of such almost the best aggregator is provided below:

In the next Section we present some numerical experiments illustrating the performance of the proposed algorithm. We will denote the MKL-predictor Equations (4), (10) constructed via linear functional strategy (LFS) in the above mentioned way by MKL-LFS.

It is interesting to compare MKL-LFS with the other two MKL-algorithms, named SimpleMKL and SpicyMKL, that have been proposed respectively in Rakotomamonjy et al. [11] and Suzuki and Tomioka [16]. We choose the first one, SimpleMKL, for comparison, because the experimental results reported in Rakotomamonjy et al. [11] show that this algorithm converges rapidly and that its efficiency compares favorably to other MKL algorithms. It is also important that a SimpleMKL toolbox based on Matlab code is available at http://www.mloss.org and, therefore, we are able to put MKL-LFS and SimpleMKL side by side. Moreover, the second competitor, SpicyMKL, uses SimpleMKL on the set-up stage.

The second algorithm, SpicyMKL, has been chosen because it is reported to be the fastest one among all other algorithms, and in particular, much faster than SimpleMKL.

To perform the comparison of SimpleMKL and SpicyMKL with MKL-LFS, we use the same experimental set-up as in Rakotomamonjy et al. [11]. More precisely, the performance evaluation is made on five data sets from the UC Irvine Machine Learning Repository: Liver, Wpbc, Ionosphere, Pima, Sonar. For each data set, the compared algorithms were run 20 times with train and test sets selected differently for each run by SimpleMKL toolbox (70% of the data set for training and 30% for testing the performance).

The SpicyMKL algorithm has been tuned in order to run its fastest version, namely with the logistic regression loss and elastic net regularization. The tuning parameters were taken the same as suggested in Suzuki and Tomioka [16].

The candidate kernels K_j, j = 1, 2, …, p are the Gaussian kernels with 10 different bandwidths, on all variables u = (u¹, u², …, u^d) and in each single variable u^j, j = 1, 2, …, d; these kernels are accompanied by the polynomial kernels of degree 1–3, again on all and on each single variable. So, the number p of kernel candidates depends on the dimension d of the input variable u = (u¹, u², …, u^d). For each of the considered data sets the value p is indicated in Table 1.

The table also reports the average values of the training time and the accuracy over 20 runs of the compared algorithms.

Since the above mentioned five data sets are associated with the classification learning task (i.e., v_i = ±1), the algorithms accuracy is measured by the percentage of the correctly classified examples (i.e., when sign(x(u)) = sign(v)) from the part of the data set that has not been used for training.

Table 1 shows that SimpleMKL and MKL-LFS are nearly identical in the performance accuracy, while SpicyMKL is less effective against this criterion. If we look at the computation time reported in Table 1, clearly, on all data sets MKL-LFS is much fasted than SimpleMKL. On the other hand, SpicyMKL is faster than two other algorithms on the sets, where the number of kernels is large, but the size of the training set is moderate. The same feature was observed in Suzuki and Tomioka [16]: the algorithm SpicaMKL is efficient when the number of unknown variables is much larger than the number of samples. It should be noted that the MKL-LFS does not suffer from this restriction.

For the purpose of comparing we follow [11] and put SimpleMKL and MKL-LFS side by side for a fixed value of the so-called hyperparameter C = 100, that corresponds to α = 1/2C = 0.005 in Equation (1). Moreover, for SpicyMKL two regularization parameters need to be selected (in our tests they are fixed as proposed in [16]). At the same time, we expect that the accuracy of MKL-LFS may be improved by combining in Equation (4) the minimizers x_j(K_j; ·) of Equation (1) corresponding to different α = α(K_j), which are properly chosen for each K = K_j, j = 1, 2, …, p. We will derive such an algorithm in the near future end expect that the speed gain observed in Table 1 for MKL-LFS allows us to implement such a posteriori choice of α effectively in time.

Moreover, our method, in principle, is not restricted to Tikhonov-type regularizations only, which is known to be affected by saturation [22]. One may potentially use any of the regularization methods, such as Landweber iteration, for example, for computing the predictors x_j(K_j, ·). In view of this, we plan to develop in the near future the MKL algorithm based on the so-called general regularization framework for learning [20].

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

The authors gratefully acknowledge the support of the Austrian Science Fund (FWF), project I1669, and of the consortium AMMODIT funded within EU H2020-MSCA-RICE.