Hinweise zum Einsatz der Google Suche
Personensuchezur unisono Personensuche
Veranstaltungssuchezur unisono Veranstaltungssuche
Katalog plus

3. Automatic Input Selection

General Idea of Input Selection


  • Forward Selection (FS)
    - fast
    - recommended for many potential inputs but only few selected
    - ignores correlations/interactions between potential inputs
  • Backward Elimination (BE)
    - slower
    - recommended if most potential inputs are selected
    - takes into account correlations/interactions between potential inputs
  • Stagewise Selection
    combination of FS and BE
  • Brute Force
    - all input combinations

Input Selection - What for?

Input Selection in the Context of Machine Learning

For choice of optimal model complexity, input selection evaluates merits of input subsets



  • Optimal bias/variance trade-off
  • Explore relevance of inputs


  • Weakening the curse of dimensionality
  • Reducing the model’s variance error
  • Increase of process understanding
  • More concise and transparent models
  • May support future design of experiments (DoE) or active learning strategies  f31_b02

Local Model Networks Enhanced

Separate Inputs for Local Models and Validity Functions

  • Whole problem split into smaller sub-problems
  • Training algorithm necessary
    - Here: Hierarchical Local Model Tree (HILOMOT)

Separation between Linear and Nonlinear Effects

  • LMNs Reveal Possibility to Separate between Linear and Nonlinear Effects
  • Any physical input ui can be included in the rule premises zi and/or in the rule consequents xi
  • The separation between linear and nonlinear effects is not possible for other types of models
  • In the following the rule premises are referred to as the z-input space, the rule consequents are referred to as the x-input space



Equivalence to Fuzzy

Interpretation as Takagi-Sugeno Fuzzy System


Input Selection With the HILOMOT-Algorithm


Input Spaces

Overview: Different Selection Strategies for Input Selection Tasks

Depending on the selection strategy either subsets or all physical inputs can be assigned to the x- and z-input space.


Demonstration Example: HILOMOT Wrapper Method


Artificial Process

  • Superposition of:
    - Hyperbola f(u1)
    - Gaussian function f(u2)
    - Linear function f(u3)
    - Normal distributed noise, σ = 0, σ = 0.05 (for u4)


  • Training samples N = 625 & N = 1296
  • Placed on a 4-dimensional grid

HILOMOT Wrapper Settings

  • Evaluation criterion: Akaike’s information criterion (AICc)
  • Search Strategy: Backward elimination (BE)

4-D Demonstration Example: Linked x-z-Selection

Results for the Linked x-z-Selection

  • Because of the BE search strategy, the results have to be read from right to left
  • Each ‘o’ is labeled with the input that is discarded in the corresponding BE step
  • A growing sample size N reduces the bad influence of the useless input u4
  • For both sample sizes N, the HILOMOT wrapper method identifies the 4th input as not useful in the linked x-z-space


4-D Demonstration Example: z-Selection

Results for the z-Selection

  • Inputs u3 and u4 are identified as useless in the z-space
    - Because the slope in u3 direction does not change, no partitioning has to be done in that direction
  • As soon as an important input in the z-space is removed the evaluation criterion gets significantly worse


4-D Demonstration Example: Separated x-z-Selection

Results for the Separated x-z-Selection

  • Input u4 is identified as useless in the x- as well as in the z-space for both sample sizes N
  • Input u3 is identified as useless in the z-space for both sample sizes N
  • After the first useful input is discarded, the evaluation criterion gets slightly worse

Real-World Demonstration Examples


Auto Miles Per Gallon (MPG) Data Set

(From the UCI Machine Learning Repository (http://archive.ics.uci.edu/ml) 2010)

  • N = 392 samples
  • q = 1 physical output:
    - Auto miles per gallon
  • p = 7 physical inputs:
    - Cylinders (u1)
    - Displacement (u2)
    - Horsepower (u3)
    - Weight (u4)
    - Acceleration (u5)
    - Model year (u6)
    - Country of origin (u7)
  • Data Splitting
    - ¾ of data: training samples (used for network training, complexity selection, input selection)
    - ¼ of data: test samples (only for final testing)
    - Heuristic, deterministic data splitting strategy

Auto MPG Wrapper Input Selection Results

Results for the x-z-Input Selection


  • Both search strategies yield similar results:
    - Best AICc-Value for 5 inputs
    - Same input combination
  • Input selection path:
    - Indicates the input combination to a given number of inputs
    - In case of BE read from right to left

Results on the Test Data

  • Mean squared error (MSE) for best AICc-Values:
    - MSE (5 inputs) = 6.0
    - MSE (all inputs) = 7.7
  • Most important inputs: u4 (car weight) and u6 (model year)
  • Improvement of model accuracy
  • Information about very important inputs

Results for the z-Input Selection


  • Best AICc-Value ES result:
    - 4 inputs
  • Best AICc-Value BE result:
    - 4 inputs
    - BUT other input combination(see input selection path)

Results on the Test Data

  • MSE for best AICc-Values:
    - MSE (best ES) = 5.2
    - MSE (best BE) = 5.3
    - MSE (all inputs) = 7.7
  • Most important inputs: u2 (displacement) and u6 (model year)
  • Improvement of model accuracy
  • Information about nonlinear input influences

Climate Control



  • 2 controlled inputs (u1, u2)
    - valve positions
  • 4 measured inputs (d1, d2, d3, d4)
    - temperature, rel. humidity, temp., temp. 
  • 2 measured outputs (y1, y2):
    - temperature, rel. humidity
  • Difficult modeling with ARX: Unstable models are possible



  • Selection on the simulated output of validation data.
  • Backward selection gives significantly better results.
  • ~15 inputs/regressors are sufficient for a good model.
  • Allowed delays: (k-1) … (k-10).
  • Forward selection: Tries to model the process with multiple nonlinear influences.
  • Backward selection: Nonlinearities only need to be considered in u1(k-1) and u2(k-2)!


Next Chapter: 4. Metamodeling     Back to Overview