A teaching learning based optimization based on orthogonal design for solving global optimization problems

In searching for optimal solutions, teaching learning based optimization (TLBO) (Rao et al. 2011a; Rao et al. 2012; Rao & Savsani 2012a) algorithms, has been shown powerful. This paper presents an, improved version of TLBO algorithm based on orthogonal design, and we call it OTLBO (Orthogonal Teaching Learning Based Optimization). OTLBO makes TLBO faster and more robust. It uses orthogonal design and generates an optimal offspring by a statistical optimal method. A new selection strategy is applied to decrease the number of generations and make the algorithm converge faster. We evaluate OTLBO to solve some benchmark function optimization problems with a large number of local minima. Simulations indicate that OTLBO is able to find the near-optimal solutions in all cases. Compared to other state-of-the-art evolutionary algorithms, OTLBO performs significantly better in terms of the quality, speed, and stability of the final solutions.


Introduction
Teaching-Learning based Optimization (TLBO) algorithm is a global optimization method originally developed by Rao et al. (Rao et al. 2011a;Rao & Savsani 2012a). It is a population-based iterative learning algorithm that exhibits some common characteristics with other evolutionary computation (EC) algorithms (Fogel 1995). However, TLBO searches for an optimum through each learner trying to achieve the experience of the teacher, which is treated as the most learned person in the society, thereby obtaining the optimum results, rather than through learners undergoing genetic operations like selection, crossover, and mutation (Shi & Eberhart 1998). Due to its simple concept and high efficiency, TLBO has become a very attractive optimization technique and has been successfully applied to many real world problems (Rao et al. 2011a;Rao & Savsani 2012a), (Rao et al. 2011b;Rao & Patel 2012;Rao & Savsani 2012b;Vedat 2012;Rao & Kalyankar 2012;Suresh Chandra & Anima 2011).
In any evolutionary algorithms the convergence rate is given prime importance for solving an optimization problem over quality of solutions. TLBO in general produces improved results in compared to other EC techniques like Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Differential Evolution (DE), and Artificial Bee Colony (ABC)(Suresh Chandra et al. 2012). However, in real-world real time applications, the major thrust is always on convergence time. To make TLBO suitable for such applications, this work focuses on improving the convergence time while without compromising the quality of results.
In our proposed work, the attempt is made to include orthogonal design method in the basic TLBO optimization. In our approach, each learner in the class of learners can be divided into several partial vectors where each of them acts as a factor in the orthogonal design. Orthogonal design is then employed to search the best scales among all the various combinations. Orthogonal design method (Fang & Ma 2001) with both orthogonal array (OA) and factor analysis (such as the statistical optimal method) is developed to sample a small, but representative set of combinations for experimentation to obtain good combinations. OA is a fractional factorial array of numbers arranged in rows and columns, where each row represents the levels of factors in each combination, and each column represents a specific factor that can be changed from each combination. It can assure a balanced comparison of levels of any factor. The term "main effect" designates the effect on response variables that one can trace to a design parameter. The array is called orthogonal because all columns can be evaluated independently of one another, and the main effect of one factor does not bother the estimation of the main effect of another factor. Recently, some researchers applied the orthogonal design method incorporated with EAs to solve optimization problems. Leung and Wang (Leung & Wang 2001) incorporated orthogonal design in genetic algorithm for numerical optimization problems found such method was more robust and statistically sound. This method was also adopted by other researchers (Ding et al. 1997;Kui-fan et al. 2002;San-You et al. 2005;Wang et al. 2007;Wang et al. 2012) to solve optimization problems. Numerical results demonstrated that these techniques had a significantly better performance than the traditional EAs on the problems studied, and the resulting algorithm can be more robust and statistically sound. In this paper, the orthogonal design is implemented on TLBO (hence called OTLBO) to make it faster and more robust. It is shown empirically that OTLBO has high performance in solving benchmark functions comprising many parameters, as compared with some existing EAs.
The rest of this paper is organized as follows. "Teaching-learning-based optimization" briefly describes TLBO as a function optimization technique and "Orthogonal design" presents some properties of the orthogonal design method. "Proposed orthogonal teachinglearning-based optimizer (OTLBO)" presents the proposed OTLBO. In "Experimental results", we test our algorithm through some benchmark functions which is followed by discussions and analysis of the optimization experiments for the OTLBO. The last section, "Conclusions and further study", is devoted to conclusions and future studies.

Teaching-learning-based optimization
This optimization method is based on the effect of the influence of a teacher on the output of learners in a class. It is a population based method and like other population based methods it uses a population of solutions to proceed to the global solution. A group of learners constitute the population in TLBO. In any optimization algorithms there are numbers of different design variables. The different design variables in TLBO are analogous to different subjects offered to learners and the learners' result is analogous to the 'fitness' , as in other population-based optimization techniques. As the teacher is considered the most learned person in the society, the best solution so far is analogous to Teacher in TLBO. The process of TLBO is divided into two parts. The first part consists of the "Teacher phase" and the second part consists of the "Learner phase". The "Teacher phase" means learning from the teacher and the "Learner phase" means learning through the interaction between learners. In the sub-sections below we briefly discuss the implementation of TLBO.

Initialization
Following are the notations used for describing the TLBO N: number of learners in class i.e. "class size" D: number of courses offered to the learners MAXIT: maximum number of allowable iterations The population X is randomly initialized by a search space bounded by matrix of N rows and D columns. The jth parameter of the ith learner is assigned values randomly using the equation where rand represents a uniformly distributed random variable within the range (0, 1), x min j and x max j represent the minimum and maximum value for jth parameter. The parameters of ith learner for the generation g are given by Teacher phase The mean parameter M g of each subject of the learners in the class at generation g is given as The learner with the minimum objective function value is considered as the teacher X g Teacher for respective iteration. The Teacher phase makes the algorithm proceed by shifting the mean of the learners towards its teacher. To obtain a new set of improved learners a random weighted differential vector is formed from the current mean and the desired mean parameters and added to the existing population of learners.
T F is the teaching factor which decides the value of mean to be changed. Value of T F can be either 1 or 2. The value of T F is decided randomly with equal probability as, Where T F is not a parameter of the TLBO algorithm. The value of T F is not given as an input to the algorithm and its value is randomly decided by the algorithm using Eq. (5). After conducting a number of experiments on many benchmark functions it is concluded that the algorithm performs better if the value of T F is between 1 and 2. However, the algorithm is found to perform much better if the value of T F is either 1 or 2 and hence to simplify the algorithm, the teaching factor is suggested to take either 1 or 2 depending on the rounding up criteria given by Eq. (5).
If Xnew g i ð Þ is found to be a superior learner than X g i ð Þ in generation g , than it replaces inferior learner X g i ð Þ in the matrix.

Learner phase
In this phase the interaction of learners with one another takes place. The process of mutual interaction tends to increase the knowledge of the learner. The random interaction among learners improves his or her knowledge. For a given learner X g i ð Þ , another learner X g r ð Þ is randomly selected (i ≠ r). The ith parameter of the matrix Xnew in the learner phase is given as

Algorithm termination
The algorithm is terminated after MAXIT iterations are completed. Details of TLBO can be refereed in (Rao et al. 2011a;Rao & Savsani 2012a).

Orthogonal design
Consider an experiment that involves some factors, each of which have several possible values called levels. Suppose that there are P factors, each factor has Q levels. The number of combinations is Q P , and for large P and Q it is not practical to evaluate all combinations.
Orthogonal design has been developed as a mathematical tool to study multi-factor and multilevel problems. It aims to extract an orthogonal array L of M rows, where each row represents a combination to be evaluated. The array has three key properties.
1) During the experiment, the array represents a subset of M combinations, from all possible Q P combinations. Computation is reduced considerably because M << Q P . 2) Each column represents a factor. If some columns are deleted from the array, it means a smaller number of factors are considered.
3) The columns of the array are orthogonal to each other. The selected subset is scattered uniformly over the search space to ensure its diversity.
A simple but efficient method is proposed in (Wing- Leung & Yuping 2001) to generate an orthogonal array L where M = Q×Q and P = Q + 1. The steps of this method are shown in Algorithm 1.

Proposed orthogonal teaching-learning-based optimizer (OTLBO)
We propose a teaching learning based optimization approach based on orthogonal design (OD). In our approach, termed OTLBO, each learner in the class of learners can be divided into several partial vectors where each of them acts as a factor in the orthogonal design. Orthogonal design is then employed to search the best scales among all the various combinations. Compared to Algorithm 1: Procedure for generating an orthogonal array L.

OD-based operator and updating strategy
Each solution to the optimization problem is represented as a learner. For an objective function with N variables, a learner is encoded in the form of where S is the population size. The standard TLBO algorithm updates the current learner by comparing with best learner (i.e. teacher) in teacher phase and with a randomly select learner in learner phase. It lacks the interaction between neighboring learners and it may easily trapped into local minima. One technique to address this problem is to employ the multi-parent crossover during evolution and this technique has been shown to improve the convergence rate when applied to GAs.
Given m learners, the question is how to execute the multi-parent efficiently. Since each learner consists of N factors, there are m N combinations. Consequently, the orthogonal design method is employed to select m (if a L m (Q P ) orthogonal array is considered, where P = N and Q = m) representative sets of combinations to shorten the computational time. The procedure of ODbased multi-parent is detailed in Algorithm 2.
Compute the fitness for all р i, j . Mix р i, j and X i, j and rank learners in the decreasing order of fitness Select the top m learners as the output.
Remark 1: The OD-based operator behaves as the local search among the selected learners.

Steps of OD-based TLBO
To obtain a more precise solution compared to the standard TLBO, the OD-based operator is employed. The elitism preservation strategy for upgrading the current population is proposed, in which the learner is updated only if its fitness is improved. The procedure for the OD-based TLBO is shown in Algorithm 3. A convergence criterion or the maximum run can be used as the termination condition.
Take the learner g as the output. Remark 2: The convergence of OTLBO is guaranteed because of the elitism preservation strategy. A learner moves only if the movement will lower this objective function.

Experimental results
We have divided our experimental works into four parts. In part1, we have done performance comparison of OTLBO against basic algorithms like PSO, DE and TLBO to establish that our proposed approach performs better than above algorithms for investigated problems. We used 20 benchmark problems in order to test the performance of the PSO, DE, TLBO and the OTLBO algorithms. This set is large enough to include many different kinds of problems such as unimodal, multimodal, regular, irregular, Algorithm 2: OD-based operator for m learners.
Algorithm 3: OD-based TLBO Table 1 Benchmark where a ¼ 0: separable, non-separable and multidimensional. For experiments in part2 to part4 few functions from these 20 functions are used and those are mentioned in the comparison tables in respective sections of the paper. Initial range, formulation, characteristics and the dimensions of these problems are listed in Table 1. Also we have used Friedman's test to determine the average ranking of algorithms based on their performance in each part. In part2 of our experiments, attempts are made to compare our proposed approach with the recent variants of PSO as per (Zhan et al. 2009;Ratnaweera et al. 2004). The results of these variants are directly taken from (Zhan et al. 2009;Ratnaweera et al. 2004) and compared with OTLBO. In part3, the performance comparisons are made with the recent variants of DE as per (Zhan et al. 2009). The part4 of our experiments devote to   For comparing the speed of the algorithms, the first thing we require is a fair time measurement. The number of iterations or generations cannot be accepted as a time measure since the algorithms perform different amount of works in their inner loops, and they have different population sizes. Hence, we choose the number of fitness function evaluations (FEs) as a measure of computation time instead of generations or iterations. Since the algorithms are stochastic in nature, the results of two successive runs usually do not match. Hence, we have taken different independent runs (with different seeds of the random number generator) of each algorithm. Numbers of FEs for different algorithms which are compared with TLBO and OTLBO are taken as in (Zhan et al. 2009;Ratnaweera et al. 2004;Alatas 2010;Zhu & Kwong 2010;Kang et al. 12;Gao & Liu 2011). However, for TLBO and OTLBO we have chosen 4.0×10 4 as maximum number of FEs. The exact numbers of FEs in which we get optimal results with TLBO and OTLBO are given in the Table 2.

Experiment 1: OTLBO vs. PSO, DE and TLBO Parameter settings
In all experiments in this section, the values of the common parameters used in each algorithm such as population size and total evaluation number were chosen to be the same. Population size was 20 and the maximum number fitness function evaluation was 4.0×10 4 for all functions. The other specific parameters of algorithms are given below: PSO Settings: Cognitive and social components c 1 ,c 2 , are constants that can be used to change the weighting between personal and population experience, respectively. In our experiments cognitive and social components were both set to 2. Inertia weight, which determines how the previous velocity of the particle influences the velocity in the next iteration, was 0.5.
DE Settings: In DE, F is a real constant which affects the differential variation between two Solutions and set to F = 0.5*(1+ rand (0, 1)) where rand (0, 1) is a uniformly distributed random number within the range [0, 1] in our experiments. Value of crossover rate, which controls the change of the diversity of the population, was chosen to be R = ( Rmax -Rmin) * (MAXIT-iter) / MAXIT where Rmax=1 and Rmin=0.5 are the maximum and minimum values of scale factor R, iter is the current iteration number and MAXIT is the maximum number of allowable iterations as recommended in (Swagatam & Ajith 2008).
TLBO Settings: For TLBO there is no such constant to set.
In Experiments 1, we compared the PSO, DE, TLBO and OTLBO algorithms on a large set of functions described in the previous section and are listed in Table 1. Each of the experiments in this section was repeated 30 times and it was terminated when it reached the maximum number of evaluations or when it reached the global minimum value with different random seeds and mean value and standard deviations of fitness value produced by the algorithms have been recorded in the Table 3 and at the same time mean value and standard deviations of no of fitness evaluation produced by the algorithms have been recorded in the Table 2.  In order to analyze the results whether there is significance between the results of each algorithm, we performed t-test on pairs of algorithms which is quite popular among researchers in evolutionary computing (Das et al. 2009). In the Table 4 we report the statistical significance level of difference of the means of PSO and OTLBO algorithm, DE and OTLBO algorithm, TLBO and OTLBO algorithm. The t value is significant at a 0.05 level of significance by two tailed test. In table '+' indicates the t value is significant' means the difference of means is not statistical siginificant and 'NA' stands for Not applicable, covering cases for which the two algorithms achieve the same accuracy results. From the Table 4, we conclude that for 15 functions OTLBO technique is performing superior then PSO and in 5 functions they are similar in performance. While comparing OTLBO and DE, we can conclude OTLBO is significantly performing better in 14 functions and in 6 cases both are similar. However, between OTLBO and TLBO, the performance comparison is competitive. OTLBO is significant only for 9 functions but both are similar for 11 functions. Table 5 reports the average ranking of optimization algorithms obtained by the Friedman's test based on the performance using Table 3. The proposed OTLBO algorithm is ranked first, followed by TLBO, DE and PSO successively.

Experiments 2: OTLBO vs. OEA, HPSO-TVAC, CLPSO and APSO
The experiments in this section constitute the comparison of the OTLBO algorithm versus OEA, HPSO-TVAC, CLPSO and APSO on 8 benchmarks described in (Zhan et al. 2009), where OEA uses the number of 3.0×10 5 FEs and HPSO-TVAC, CLPSO and APSO use the number of 2.0×10 5 FEs, where as OTLBO runs for 4.0×10 4 FEs. The results of OEA, HPSO-TVAC, CLPSO and APSO are gained from (Zhan et al. 2009;Ratnaweera et al. 2004) directly. In the last column of Table 6 shows the siginificance level between best and second best algorithm using t test at a 0.05 level of significance by two tailed test. Note that here '+' indicates the t value is significant means the difference of means is not statistical siginificant and 'NA' stands for Not applicable, covering cases for which the two algorithms achieve the same accuracy results. As can be seen from Table 6, OTLBO greatly outperforms OEA, HPSO-TVAC, CLPSO and APSO with better mean and standard deviation. Table 7 reports the average ranking of optimization algorithms obtained by the Friedman's test based on the performance using Table 6. The proposed OTLBO algorithm is ranked first, followed by APSO,OEA, HPSO, CLPSO successively.

Experiment 3: OTLBO vs. JADE, jDE, SaDE, CoDE, EPSDE
The experiments in this section constitute the comparison of the OTLBO algorithm versus SaDE, jDE, JADE,      (Zhan et al. 2009). The results of JADE, jDE and SaDE are gained from (Zhan et al. 2009) directly. In the last column of Table 8 shows the significance level between best and second best algorithm using t test at a 0.05 level of significance by two tailed test. Note that here '+' indicates the t value is significant, '.' means the difference of means is not statistical significant and 'NA' stands for Not applicable, covering cases for which the two algorithms achieve the same accuracy results It can be seen from Table 8 that OTLBO performs much better than these DE variants on almost all the functions. Table 9 reports the average ranking of optimization algorithms obtained by the Friedman's test based on the performance using Table 8. The proposed OTLBO algorithm is ranked first, followed by EPSDE, JADE, CoDE, jDE, SaDE successively. The parameters of the algorithms are identical to . In the last column of Table 10 shows the significance level between best and second best algorithm using t test at a 0.05 level of significance by two tailed test. Note that here '+' indicates the t value is significant means the difference of means is not statistical significant and 'NA' stands for Not applicable, covering cases for which the two algorithms achieve the same accuracy results The results, which have been summarized in Table 10, show that OTLBO performs much better in most cases than these ABC variants. Table 11 reports the average ranking of optimization algorithms obtained by the Friedman's test based on the performance using Table 10. The proposed OTLBO algorithm is ranked first, followed by IABC, GABC, RABC, CABC successively.

Conclusions and further study
In this work orthogonal design approach is implemented to optimize global benchmark functions using basic Teaching-Learning based Optimization (TLBO). The proposed approach is known as Orthogonal TLBO (OTLBO).The benefit is derived in making the basic TLBO fasters with our proposed approach. Orthogonal design makes the search efficient in a large sample space to arrive at optimum solution. The paper discusses the fundamentals of orthogonal design and its framework in TLBO. The performance comparisons are done with TLBO and other evolutionary computation techniques like particle swarm optimization (PSO), Differential evolution (DE), artificial bee colony (ABC) and several of variants of these algorithms suggested by other researchers. From the results analysis it is evident that OTLBO outperforms all other approaches including basic TLBO for all benchmark functions investigated in our work. The efficiency of the proposed approach is compared with other algorithms in terms of number of function evaluations (FEs). We can conclude by saying that OTLBO is a very powerful approach of optimizing different types of problems which are separable, nonseparable, unimodal and multimodal in providing quality optimum results in faster convergence time compared to very popular evolutionary techniques like PSO, DE, ABC and its variants. As a further research it remains to be seen how this adapts to multi-objective optimization problems and also some engineering applications from mechanical, chemical or data mining may be investigated.