I recently ran into a similar problem: How to manage extracting a variety of features from a large dataset, without knowing up front what all of them would be. (Even calculating mean values repeatedly would be computationally expensive.) Further, how would I manage predictions based on different feature sets? Meaning, if I added a new feature, how would I know which models to train on new features? It could quickly snowball into a huge mess.
My current solution is to track it all in a local NoSQL database (MongoDB). For example, I might have a collection features, each entry of which has a name, a description of how the feature was was calculated, the python file that ran the extraction, etc.
Likewise, a collection models includes models run on the data. Each entry might have a name, a list of features that were used to train the model, its eventual parameters, predicted values on a held-out test set, metrics for how the model performed, etc.
From my vantage point, this has a number of benefits:
- By saving predictions, I can use them later in ensemble predictions.
- Because I keep track of which features were used, I know which ones need retraining as I extract more features.
- By saving model descriptions, I ensure that I always know what I've tried. I never have to wonder, "Have I tried LASSO with regularization parameters set by grid-search CV?" I can always look it up, and see how successful it was.
From your question, it sounds like you could adapt this approach to your problem's workflow. Install Mongo or another database of choice, and then save each experimental run, its inputs, its results, and anything else you might wish to track over the course of the project. This should be much easier to query than a spreadsheet, at the least.
