Evaluation metrics matter: predicting sentiment from financial news headlines
Andrew Moore
SCC, Lancaster University
The SemEval Task 5 track 2 competition this year was to predict the sentiment with respect to a company in financial news headlines. We came 5th out of 45 competitors using a Bi-Directional Long Short-Term Memory (LSTM) deep learning model. We describe this approach and how we practically implemented it and the other methods we attempted. We describe the experience of participating in the event and how the evaluation metric used to evaluate the models should reflect the real world problem being solved. This talk we hope will be interesting to all wanting to see how easy it is now to implement deep learning models and how we can learn from others mistakes with regards to setting up an evaluation task to solve a real world problem.