Trjtdtk

Let's say I have a data set but I don't know what features are relevant to solve a classification/regression problem.

In this case, is it worth/good to use a dimension reduction algorithm and then apply a classification algorithm? Or can I just select "randomly" my features by using my common sense and then try to tune my algorithm next?

Also if someone has some explanation of a dimension reduction "in real life with real use case" it would be great because I feel my comprehension of dimension reduction is wrong!

Well, let's say it depends on the distribution of your data. In approaches like PCA the approach does not care about the labels of the data in hand. This is why PCA may lead to data which are sometimes difficult to be separated or vice versa. PCA just cares about which direction leads to more variance and take that direction as a new basis. Not caring about the labels is why you cannot say it may lead to a better space for classification or not. You have to employ that and after that, investigate whether it's helpful or not. Approaches like LDA or other variants of that take care of the labels but they are linear classifiers which are not strong at least in a current feature space where you've not done any feature engineering.

The question is: why you want to apply a features selection?

In many algorithms, you can use all the features and it will be the model that picks the one that are more important for the prediction.

To me some reasons to apply features selection is:

• business cost of using more features


• interpretation of results


• fear that noise in the data can let the model pick up wrong features and bias results

If you don't care which features are included, using PCA (or something similar) can help.

If you do have some information on which features influence classification or regression, you can certainly try to fit a model without dimensional reduction.

PCA, which is one of the more common dimensional reduction techniques, yields vectors that are all orthogonal (as in, uncorrelated). This means that even if your features are correlated, after the dimensional reduction, your model won't struggle with collinearity. Depending on your model type, this can be crucial. A real life example could be any housing dataset, where the features describe the house and the target is the price. Many of the features will be correlated (e.g. number of bathrooms and number of bedroom or number of rooms and square footage), and so a linear regression model may get tripped up by the collinearity. Dimensional reduction will capture the variance across the features while yielding fewer columns.

For feature selection, we can also use Random Forest. Check this one:
https://chrisalbon.com/machine_learning/trees_and_forests/feature_selection_using_random_forest/

Also, forward/backward stepwise variable selection is an option. Check this one:
https://gerardnico.com/data_mining/stepwise_regression