Trjtdtk

i'm trying to apply scikit learns decision tree on the following dataset with the goal of classifying the data:

sensordata:

• multiple .csv files


• every .csv file has multiple sensors (see here)


• each .csv file has one label (0 or 1)

So far I've tried to train my model with Pandas Series. It worked, but the decision tree couldn't differate the features/sensors. Is pandas series the right approach for analyse data like this? Or does anyone have another solution for this problem?

For usage you need to flatten the 2D raw sensor data into 1D features. Below code demonstrates the basics.

What kind of feature engineering to apply for best predictive effect depends entirely on the nature of your sensors and problem. There are no details about this in the question or data provided.

Feature Engineering

The overall process is:

1. Look for patterns in the data (Exploratory Data Analysis)


2. Attempt to create a new feature which describes this pattern


3. Evaluate the new set of features using cross-validation


4. Analyze the samples that your classifier got wrong (Error Analysis)


5. Repeat from 1) until performance is good enough

Here are some things you should try:

• Plot the raw sensor data from a few samples of the positive and negative class.




• Plot the distributions (histogram) for each class of each raw sensor values across the entire dataset.




• Try to standardize the data. For each time-series of sensor data, remove the mean and divide by the standard deviation for each sample.




• Try some standard statistical summarizations on each time-series. Max, min, mean, std, skew, kurtosis. Unlikely to be better than something tailored to the patterns you see, but sometimes performs OK.

Focus first on uni-variate features per sensor. The decision-tree will be good at combining these together.

Fitting to classifier

import numpy
import pandas
from sklearn.ensemble import RandomForestClassifier

def get_sensor_data():

 timesteps = 10
 times = numpy.linspace(0.1, 1.0, timesteps)
 df = pandas.DataFrame(
 'time': times,
 'sensor1': numpy.random.random(timesteps),
 'sensor2': numpy.random.random(timesteps),
 'sensor3': numpy.random.random(timesteps),
 'sensor4': numpy.random.random(timesteps), 
 )

 return df

samples = [ get_sensor_data() for _ in range(100) ]
labels = [ int(numpy.random.random() > 0.5) for _ in range(100) ]
assert len(samples) == len(labels)

print('sample from CSV file:n', samples[0], 'nlabel', labels[0], 'n')

def to_features(data):
 # remove time column
 feature_columns = list(set(data.columns) - set(['time']))
 # TODO: do smarter feature engineering here
 sensor_values = data[feature_columns].values 
 # Note: the features must be 1D for scikit-learn classifiers
 features = sensor_values.flatten()
 assert len(features.shape) == 1, features.shape
 return features

features = numpy.stack([ to_features(d) for d in samples ])

assert features.shape[0] == len(samples)
print('Features:', features.shape, 'n', features[0])

# XXX: do train/test splits etc
est = RandomForestClassifier(n_estimators=10, min_samples_leaf=0.01)
est.fit(features, labels)

Example output

sample from CSV file:
 time sensor1 sensor2 sensor3 sensor4
0 0.1 0.820667 0.346542 0.625512 0.774050
1 0.2 0.821934 0.241652 0.485608 0.188131
2 0.3 0.264697 0.780841 0.137018 0.117096
3 0.4 0.464143 0.457126 0.972894 0.600710
4 0.5 0.530302 0.027401 0.876191 0.563788
5 0.6 0.598231 0.291814 0.588032 0.143753
6 0.7 0.627435 0.036549 0.276131 0.311099
7 0.8 0.527908 0.197046 0.580293 0.123796
8 0.9 0.068682 0.880533 0.956394 0.787993
9 1.0 0.244478 0.306716 0.586049 0.373013 
label 1 

Features: (100, 40) 
 [0.82066682 0.62551234 0.77405 0.34654243 0.82193414 0.48560828
 0.18813108 0.24165186 0.26469686 0.1370181 0.11709553 0.78084136
 0.46414318 0.97289382 0.60070974 0.45712632 0.53030219 0.8761905
 0.5637877 0.02740072 0.59823073 0.58803188 0.14375282 0.29181434
 0.62743516 0.27613083 0.31109894 0.03654882 0.52790773 0.58029298
 0.1237963 0.19704597 0.06868206 0.95639405 0.78799333 0.88053276
 0.24447754 0.5860489 0.37301339 0.30671624]

```