Trjtdtk

I have a non linear data set, and I am using SVM (RBF kernel) to build a classification model, but not sure how to set the best hyperparameters of the SVM, C and gamma in Matlab fitcsvm. And is it acceptable if it is trial and error approach setting arbitrary values, until finding the best performance?

Sadly, there is no easy solution for hyperparameter tuning. Basically, you have two options:

1. 
   Manual Adjustment: Read the documentation of the SVM and dive into the corresponding literature. Try to understand how the different kernel functions work and which functions are appropriate for your classification problem. Set the hyperparameters to the best of your knowledge and then observe what difference slight deviations make. Of course, there will still be some degree of trial and error in this approach.


2. 
   Grid Search: For all categoric parameters (e.g. kernel function), set up a list of all possible categories and for all metric parameters (e.g. epsilon), define a range of reasonable values. This is your hyperparameter space. Subsequently, run a multitude of SVMs with randomly selected parameters from the hyperparameter space. In the end, you choose the hyperparameter combination with the highest accuracy. (Note: there are also other approaches with more sophisticated methods than random selection)
   

Both methods are widely used, however, if you are not very familiar with SVM's, I highly recommend the manual adjustment.

Of course, it is very tempting to simply set up a grid search, grab something to eat and later pick the best performing hyperparameter combination. But in the end, you will not learn anything and you will not know if these hyperparameters are a reasonable choice for your problem. Broadening your knowledge will be beneficial in the future, both because it reduces the effort for future manual adjustments and because in the future you will be able to narrow down the hyperparameter space for the grid search, which will significantly reduce computing time.

Well, there is a bunch of articles that tries to tackle this problem but basically, to guarantee a good solution you will need to do Grid Search (sklearn tutorial on it)

You can use various techniques for that, for example:

• Binary Gridsearch: Try the maximum and the minimum value in the interval you want, and the middle point, check which side is best and set the minimum or the maximum as your previous middle point according.

That can help you find an average solution fast, but remember that this comes without warranty.

• Exhaustive Grid Search




• Randomized Parameter Optimization




• Genetic Algorithms

Check the link for sklearn for more information. I research on kernel methods and this is aways a bit annoying thing to tackle.

Note: If you're only going to try RBF I would advise to use $sigma$ as a value betwen the norm of your training samples $pm 80%$. You could also normalize you vectors to norm 1 and limit the search to $0.2$ to $1.8$. Usually too small $sigma$ gives high sensibility to noise while too big aproximates a straight line (loses non-linear power)