stocksight

Stock market analyzer and predictor using Elasticsearch, Twitter, News headlines and Python natural language processing and sentiment analysis

shirosaidev

2164

467

Python

stocksight

Stock market analyzer and stock predictor using Elasticsearch, Twitter, News headlines and Python natural language processing and sentiment analysis. How much do emotions on Twitter and news headlines affect a stock’s price? Let’s find out…

About

stocksight is an open source stock market analysis software that uses Elasticsearch to store Twitter and news headlines data for stocks. stocksight analyzes the emotions of what the author writes and does sentiment analysis on the text to determine how the author “feels” about a stock. It could be used for more than finding sentiment of just stocks, it could be used to find sentiment of anything…

Slack workspace

Join the conversation, get support, etc on stocksight Slack.

Requirements

Python 3.x
Elasticsearch 5.x
Kibana 5.x
elasticsearch python module
nltk python module
requests python module
tweepy python module
beautifulsoup4 python module
textblob python module
vaderSentiment python module
newspaper3k python module

Download

$ git clone https://github.com/shirosaidev/stocksight.git
$ cd stocksight

Download latest version

Screenshot

Stocksight Kibana dashboard

Install - Docker

*** See how to use below before building the Docker containers ***

Download/clone stocksight repo with git.
Set up stocksight, elasticsearch and kibana containers using Docker compose

cd stocksight
cp config.py.sample config.py
***see how to use below for config.py (stocksight config) changes***
docker-compose build && docker-compose up

This will volume mount config.py (stocksight settings) and twitteruserids.txt to those files in your local git cloned “stocksight” directory

Once all the containers have started up, shell into the container

docker exec -it stocksight_stocksight_1 bash

See examples below for running stocksight.

Install - local

Recommended to install Elasticsearch and Kibana in local machine or other machine/vm/docker

Install python requirements using pip

pip install -r requirements.txt

Install python nltk data

python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords')"

How to use

Create a new twitter application and generate your consumer key and access token. https://developer.twitter.com/en/docs/basics/developer-portal/guides/apps.html
https://developer.twitter.com/en/docs/basics/authentication/guides/access-tokens.html
Copy config.py.sample to config.py (stocksight config file)
Set elasticsearch settings in config.py for your env (for Docker, set elasticsearch_host = "elasticsearch")
Add twitter consumer key/access token and secrets to config.py
Edit config.py and modify NLTK tokens required/ignored and twitter feeds you want to mine. NLTK tokens required are keywords which must be in tweet before adding it to Elasticsearch (whitelist). NLTK tokens ignored are keywords which if are found in tweet, it will not be added to Elasticsearch (blacklist).

Examples

Run sentiment.py to create ‘stocksight’ index in Elasticsearch and start mining and analyzing Tweets using keywords and the stock symbol TSLA

$ python sentiment.py -s TSLA -k 'Elon Musk',Musk,Tesla,SpaceX --debug

Start mining and analyzing Tweets using keywords and the stock symbol TSLA and follow any url links in tweets and performing sentiment analysis on the link web page as well as the tweet

$ python sentiment.py -s TSLA -k 'Elon Musk',Musk,Tesla,SpaceX -l --debug

Start mining and analyzing Tweets from feeds in config using cached user ids from file (if you change any of the twitter feeds in the config file, you need to delete this file and recreate it without -f)

$ python sentiment.py -s TSLA -f twitteruserids.txt --debug

Start mining and analyzing News headlines and following headline links and scraping relevant text on landing page

$ python sentiment.py -s TSLA --followlinks --debug

Run stockprice.py to add stock prices to ‘stocksight’ index in Elasticsearch

$ python stockprice.py -s TSLA --debug

Kibana

Load ‘stocksight’ index in Kibana. For index pattern you can use ‘stocksight’ if you only have the single index or ‘stocksight-*’, etc. For time-field name you will want to use the date/time field ‘date’.

To import the saved exported visualizations/dashboard, go to Kibana, click on management, click on saved objects, click on the import button and import the export.json file.

CLI options

usage: sentiment.py [-h] [-i INDEX] [-d] -s SYMBOL [-k KEYWORDS] [-a] [-u URL]
                    [-f FILE] [-l] [-n] [--frequency FREQUENCY]
                    [--followlinks] [-w]
                    [--overridetokensreq TOKEN [TOKEN ...]]
                    [--overridetokensignore TOKEN [TOKEN ...]] [-v] [--debug]
                    [-q] [-V]

optional arguments:
  -h, --help            show this help message and exit
  -i INDEX, --index INDEX
                        Index name for Elasticsearch (default: stocksight)
  -d, --delindex        Delete existing Elasticsearch index first
  -s SYMBOL, --symbol SYMBOL
                        Stock symbol you are interesed in searching for,
                        example: TSLA
  -k KEYWORDS, --keywords KEYWORDS
                        Use keywords to search for in Tweets instead of feeds.
                        Separated by comma, case insensitive, spaces are ANDs
                        commas are ORs. Example: TSLA,'Elon
                        Musk',Musk,Tesla,SpaceX
  -a, --addtokens       Add nltk tokens required from config to keywords
  -u URL, --url URL     Use twitter users from any links in web page at url
  -f FILE, --file FILE  Use twitter user ids from file
  -l, --linksentiment   Follow any link url in tweets and analyze sentiment on
                        web page
  -n, --newsheadlines   Get news headlines instead of Twitter using stock
                        symbol from -s
  --frequency FREQUENCY
                        How often in seconds to retrieve news headlines
                        (default: 120 sec)
  --followlinks         Follow links on news headlines and scrape relevant
                        text from landing page
  -w, --websentiment    Get sentiment results from text processing website
  --overridetokensreq TOKEN [TOKEN ...]
                        Override nltk required tokens from config, separate
                        with space
  --overridetokensignore TOKEN [TOKEN ...]
                        Override nltk ignore tokens from config, separate with
                        space
  -v, --verbose         Increase output verbosity
  --debug               Debug message output
  -q, --quiet           Run quiet with no message output
  -V, --version         Prints version and exits
  
  
usage: stockprice.py [-h] [-i INDEX] [-d] [-s SYMBOL] [-f FREQUENCY] [-v]
                     [--debug] [-q] [-V]

optional arguments:
  -h, --help            show this help message and exit
  -i INDEX, --index INDEX
                        Index name for Elasticsearch (default: stocksight)
  -d, --delindex        Delete existing Elasticsearch index first
  -s SYMBOL, --symbol SYMBOL
                        Stock symbol to use, example: TSLA
  -f FREQUENCY, --frequency FREQUENCY
                        How often in seconds to retrieve stock data (default:
                        120 sec)
  -v, --verbose         Increase output verbosity
  --debug               Debug message output
  -q, --quiet           Run quiet with no message output
  -V, --version         Prints version and exits

Disclaimer

This software is for educational purposes only. USE THE SOFTWARE AT YOUR OWN RISK. THE AUTHORS AND ALL AFFILIATES ASSUME NO RESPONSIBILITY FOR YOUR TRADING RESULTS. Do not risk money which you are afraid to lose. There might be bugs in the code - this software DOES NOT come with ANY warranty.