SentimentTrader

Online financial discussion moves quickly, but it is also extremely messy. People use ticker symbols, company names, abbreviations, jokes, sarcasm, and everyday words that can easily be mistaken for stocks.

A ticker such as AI or IT might refer to a company, or it might just be part of a normal sentence. Even when a post clearly discusses a stock, not every mention should be treated equally.

Reddit activity, Google search interest, and stock prices all arrive in different formats and at different intervals, so the system has to normalize each source and align the data around the same ticker symbols and dates.

I treated the historical dataset as the primary product instead of presenting the first version as a stock-prediction system.

That made the project more honest. Financial markets are noisy, and a small dataset can make a model look much more accurate than it really is.

I also kept the output structured so different sentiment models, engagement signals, search interest, and market features can be compared or combined later instead of locking the system into one interpretation too early.

Python, Pandas, and NumPy handle most of the pipeline because the core work is text processing, feature creation, date alignment, aggregation, and dataset management.

The Reddit portion uses PRAW to collect finance discussions, then cleans text, extracts possible ticker mentions, and filters out common false positives before sentiment analysis.

The sentiment layer uses financial and social-media models, including FinBERT and Twitter-RoBERTa, so the system can compare formal financial language with the shorter, more informal language common in Reddit comments.

The final pipeline combines Reddit sentiment, engagement signals, Google Trends data, and market data into a repeated data-collection workflow that continues growing the dataset over time.

Ticker extraction was one of the first major challenges because a simple regular expression creates many incorrect matches from normal words, unsupported symbols, and ambiguous references.

Another challenge was deciding what a sentiment score should represent. A single positive post should not necessarily outweigh a large negative discussion, but raw engagement can also be misleading.

Timing was also difficult because a post made after the market closes should not be treated the same as one made before trading begins, and Reddit, Google Trends, and market prices do not naturally share the same reporting windows.

SentimentTrader is currently operating as a data-collection and feature-engineering pipeline.

The pipeline handles external APIs, natural-language processing, scheduled data workflows, noisy real-world datasets, and financial time-series data as one running system, built within the practical limits of machine-learning projects.

The project is now accumulating historical observations that can later be divided into training, validation, and out-of-sample testing periods.

The next phase is to begin testing prediction models once the dataset covers enough time and different market conditions.

The first models will likely focus on limited questions, such as whether changes in discussion volume or sentiment provide any useful signal for short-term volatility or directional movement.

I also want to improve company-mention detection, experiment with time-of-day features, monitor changes in sentiment rather than only daily averages, and build a dashboard for inspecting individual tickers and pipeline health.