Multimodal Deep Learning for Financial Market Sentiment Prediction Using Geospatial Data and Location-Based Social Media Signals
Keywords:
multimodal deep learning; financial sentiment; geotagged social media; spatial attention; uncertainty calibration; market predictionAbstract
Financial market sentiment models increasingly ingest social media, yet most treat posts as aspatial text streams. This misses two facts: social media is location-based, and local context conditions whether an expressed attitude is economically informative. This manuscript develops a multimodal deep learning framework for financial market sentiment prediction that fuses text, venue and street-level imagery, geospatial graph structure, and market variables. The contribution is a formal architecture and analytical treatment. Modality-specific encoders represent language, visual scene, place, and market state; a geographic kernel converts irregular geotagged posts into smooth local signals; graph attention propagates information across places; a temporal Transformer aligns asynchronous streams; and a calibrated predictive distribution supports uncertainty-aware decisions. We specify loss terms for prediction, calibration, spatial regularization, and complexity control, and we connect decision rules to estimated mean and uncertainty. The analysis identifies key design conditions: spatial weighting must avoid neighborhood confounding; image and text fusion must tolerate missing modalities; temporal splits must prevent look-ahead leakage; and calibration must be evaluated separately from discrimination. It also yields testable implications about when geospatial signals should improve sentiment prediction: when local participation is dense, venue imagery is informative, spatial autocorrelation is strong, and posts are less bot-contaminated. The framework allows researchers to compare text-only, aspatial multimodal, and spatially grounded models under a common set of estimands. Practically, it supports risk-aware interpretation of social sentiment and clarifies the evidence required before deployment in trading, surveillance, or market monitoring.
References
1. Bollen, J., Mao, H., & Zeng, X. (2011). Twitter mood predicts the stock market. Journal of Computational Science, 2(1), 1–8.
2. Tetlock, P. C. (2007). Giving content to investor sentiment: The role of media in the stock market. Journal of Finance, 62(3), 1139–1168.
3. Loughran, T., & McDonald, B. (2011). When is a liability not a liability? Textual analysis, dictionaries, and 10-Ks. Journal of Finance, 66(1), 35–65.
4. Kearney, C., & Liu, S. (2014). Textual sentiment in finance: A survey of methods and models. International Journal of Forecasting, 30(3), 643–658.
5. Chen, H., De, P., Hu, Y., & Hwang, B.-H. (2014). Wisdom of crowds: The value of stock opinions transmitted through social media. Review of Financial Studies, 27(5), 1367–1403.
6. Fischer, T., & Krauss, C. (2018). Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research, 270(2), 654–669.
7. Gu, S., Kelly, B., & Xiu, D. (2020). Empirical asset pricing via machine learning. The Review of Financial Studies, 33(5), 2223–2273.
8. Aman, J., Matisziw, T. C., Kim, J. B., & Luo, D. (2022, April). Sensing the city: Leveraging geotagged social media posts and street view imagery to model urban streetscapes using deep neural networks. In Proceedings of the International Conference on Computer-Aided Architectural Design Research in Asia (pp. 595-604). Association for Computer-Aided Architectural Design Research in Asia.
9. Cheng, Z., Caverlee, J., & Lee, K. (2010). You are where you tweet: A content-based approach to geo-locating Twitter users. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (pp. 759–768). ACM.
10. Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019). Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 423–443.
11. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (Vol. 30, pp. 5998–6008).
12. Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019). Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 6558–6569).
13. Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2017). Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (Vol. 30, pp. 6402–6413).
14. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (Vol. 70, pp. 1321–1330). PMLR.
15. Angelopoulos, A. N., & Bates, S. (2023). Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4), 494–591.
16. Anselin, L. (1995). Local indicators of spatial association—LISA. Geographical Analysis, 27(2), 93–115.
17. Tobler, W. R. (1970). A computer movie simulating urban growth in the Detroit region. Economic Geography, 46(sup1), 234–240.
18. Getis, A., & Ord, J. K. (1992). The analysis of spatial association by use of distance statistics. Geographical Analysis, 24(3), 189–206.
19. Hagenau, M., Liebmann, M., & Neumann, D. (2013). Automated news reading: Stock price prediction based on financial news using context-capturing features. Decision Support Systems, 55(3), 685–697.
20. Renault, T. (2017). Intraday online investor sentiment and return patterns in the U.S. stock market. Journal of Banking & Finance, 84, 25–40.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Advanced Financial Research

This work is licensed under a Creative Commons Attribution 4.0 International License.