Text Mining

UMN has trial access to a full slate of text mining workshops in Python via Constellate. Login on campus or use the VPN from off-campus.The trial ends on July 1, 2025, when JSTOR plans to sunset the Constellate platform.

Access and perform text analysis of scholarly books and articles from JSTOR and Portico. Also includes the large historical newspaper digital archive Chronicling America.

The General Index is an open-access collection of n-grams (words and phrases) and metadata from over 100 million journal articles to support text mining. Articles include both open access and paywalled content, available here as derived data (not human-readable text).

The LDC collects language data from both written texts and transcriptions of speech, in various languages, to support corpus linguistics. The Library subscription, and access to LDC datasets, begins in 2022 though some older datasets are also available.

Over four million articles from full-text biomedical and life sciences journal articles in PubMed Central are available in XML and plain text formats via Amazon Web Services.

Many (but not all) of the digital archives that we subscribe to via Readex are available to explore using Voyant Tools. After searching a database, select the results of interest and choose the "Explore" button to launch an interface to allow you to analyze and visualize the texts. More info from Readex.

Nearly 14 million books from the HathiTrust Library are currently available for analysis, offering various levels of immediate access. Check the HTRC tab on this guide for more information to help you get started.

Project Gutenberg hosts over 50k ebooks, most of which are older books in the public domain. If you want to download more than about 100 books/day, use one of the mirror sites listed from the link above.

Thousands of titles from academic publishers, primarily from 2000 and 2017. The open access ebooks dataset includes full-text OCR and title-level metadata.

The full digital edition of the Encyclopaedia Britannica from 1768-1860, available for bulk download in XML, image files, and/or plain-text. See other digital text collections from the National Library of Scotland.

UMN has trial access to a full slate of text mining workshops in Python via Constellate. Login on campus or use the VPN from off-campus.The trial ends on July 1, 2025, when JSTOR plans to sunset the Constellate platform.

Access and perform text analysis of scholarly books and articles from JSTOR and Portico. Also includes the large historical newspaper digital archive Chronicling America.

Archival collections via Adam Matthew are available for text mining by request. OCR full text and metadata from collections such as American Indian Newspapers, Apartheid South Africa, and many more can be downloaded in XML or JSON formats to support digital scholarship projects.

Textual datasets and bibliometric data from a variety of Canadian sources including hundreds of Érudit journals, archives from a variety of national and provincial collections from the 1800s to present, parliamentary debates, and more.

The Chronicling America API provides access to information about historic U.S. newspapers and millions of digitized newspaper pages and their OCR data is available for bulk download. See the full list of digitized newspaper titles (1836-1922) for more information.

The datasets used for a deep learning project includes 90,000 CNN articles and over 190,000 Daily Mail articles downloaded from the Wayback Machine and available for bulk download.

The White House recently issued a call to action for text and data mining researchers to work with the COVID-19 Open Research Dataset (CORD-19) of scholarly literature about COVID-19, SARS-CoV-2, and the Coronavirus group.

The GDELT Project is a realtime network diagram and database of global human society for open research. Coded data is available as raw downloads of via Google BigQuery, including collections on news, television, images, books, academic literature and the open web.

Three online tools—Explorer, Topic Mapper, and Source Manager—to analyze stories and topics across print, broadcast and digital news collections. Developed by the MIT Center for Civic Media and the Berkman Klein Center for Internet & Society at Harvard. Data is also available via API and as open source software.

API service that allows you to query online news sources from the past month including major publications such as the New York Times, ABC News, and Al Jazeera. Register for a free API key to get started.

The Article Search API provides access to headlines, abstracts, lead paragraphs and more (but NOT full-text articles) from the New York Times, 1851 to present.

Many (but not all) of the digital archives that we subscribe to via Readex are available to explore using Voyant Tools. After searching a database, select the results of interest and choose the "Explore" button to launch an interface to allow you to analyze and visualize the texts. More info from Readex.

Free non-commercial API key to access full-text and metadata from dozens of Australian newspapers and other sources, primarily historical.

Full-text and metadata bulk downloads for open access scholarship in Physics, Mathematics, Computer Science, Quantitative Biology, Quantitative Finance, Statistics, Electrical Engineering and Systems Science, and Economics.

BMJ allow text and data mining following their terms of service (see link) using the CrossRef APIs.

Text mining is allowed, but no API is provided. To access a large amount of content for a text and data mining project, please do not scrape the site, but contact openresearch@cambridge.org for help.

CSP grants subscribers the right to text and data mine (TDM) online content for non-commercial purposes. Follow terms and access content via CrossRef API.

UMN has trial access to a full slate of text mining workshops in Python via Constellate. Login on campus or use the VPN from off-campus.The trial ends on July 1, 2025, when JSTOR plans to sunset the Constellate platform.

Access and perform text analysis of scholarly books and articles from JSTOR and Portico. Also includes the large historical newspaper digital archive Chronicling America.

CORE provides a central API to access full content from tens of thousands of openly available scientific publications from thousands of OA repositories. They also provide full datasets by request.

Researchers can text mine UMN-subscribed journals and books published by Elsevier on the ScienceDirect full-text platform. Sign up for a developer account to use the Elsevier APIs for non-commercial purposes. Query the API from University of Minnesota, Twin Cities IP ranges to ensure full access. You can also use the APIs to access citation data and abstracts from scholarly journals indexed by Scopus. For more information, see their Text Mining documentation.

The General Index is an open-access collection of n-grams (words and phrases) and metadata from over 100 million journal articles to support text mining. Articles include both open access and paywalled content, available here as derived data (not human-readable text).

Register for a free account to get an API key to query content from IEEE Xplore. Data responses include URLs for full-text HTML and PDFs and IEEE allows text mining for non-commercial research purposes.

Data for Research allows you to download word frequencies, citations, key terms, and ngrams for up to 25,000 JSTOR articles at a time, or to easily submit requests for larger sets of articles. See also: JSTOR DfR in GitHub - A number of Python and R packages to work with JSTOR DfR data.JSTOR's Text Analyzer, a reverse search engine that analyzes documents that you upload (your own, or other articles) to find related materials in JSTOR.Public domain and OA datasets include full OCR text from early journals and current academic press open access titles.

OUP accommodates TDM for non-commercial use. E-mail Data.Mining@oup.com to arrange for access.

Python tool for downloading/updating/maintaining a repository of all PLOS XML article files. Use this program to download all PLOS XML article files instead of doing web scraping. See also: PLOS APIs to query content from the seven open-access peer-reviewed journals from the Public Library of Science using any of the twenty-three terms in the PLOS Search.

Over four million articles from full-text biomedical and life sciences journal articles in PubMed Central are available in XML and plain text formats via Amazon Web Services.

Text and data mining for non-commercial purposes is allowed. You can systematically download articles available via UMN subscriptions as long as you follow their posted request limits and agree to their terms (from link above).

Open online Jupyter book containing short scholarly API code examples for how to work with various scholarly web service APIs such as ScienceDirect, Crossref, Scopus, Chronicling America, and more.

"Individual researchers are encouraged to download subscription (and open access) journal articles and books for TDM purposes directly from Springer Nature’s content platforms. The selection of desired articles can be conducted by using existing search methods and tools, such as PubMed, Web of Science, or Springer Nature’s Metadata API, among others. An API key is necessary only if researchers want to use Springer Nature’s TDM APIs." via Springer's Text and Data Mining Policy.

The REST API gives anyone free, programmatic access to the Unpaywall database, which includes Open Access content from over 50,000 publishers and repositories. The database harvests content from open indexes including DOAJ and Crossref, as well as specific OA publishers.

Academic subscribers can access subscribed content for text and data mining once they have accepted the Wiley click-through TDM license and received an API token.

Search and retrieve metadata from over 13,000 interdisciplinary open access journals in DOAJ.

Researchers can text mine UMN-subscribed journals and books published by Elsevier on the ScienceDirect full-text platform. Sign up for a developer account to use the Elsevier APIs for non-commercial purposes. Query the API from University of Minnesota, Twin Cities IP ranges to ensure full access. You can also use the APIs to access citation data and abstracts from scholarly journals indexed by Scopus. For more information, see their Text Mining documentation.

The American Medical Association network of journals provides individuals who have access via an institutional site-license (such as those at UMN) the ability to download aggregated metadata of subscribed content. Sign up for a free TDM account to access bulk downloads of JSON files.

The LOC release of 25 million MARC records for free bulk download. MARC (Machine Readable Cataloging Records) is a international metadata standard for the representation and communication of bibliographic and related information.

The Article Search API provides access to headlines, abstracts, lead paragraphs and more (but NOT full-text articles) from the New York Times, 1851 to present.

Downloadable datasets for citations drawn from two large academic graphs: Microsoft Academic Graph (MAG) and AMiner.

An open and free API and data snapshot of scholarly citations linking works, authors, venues, institutions, and concepts. Includes data from the Microsoft Academic Graph, Crossref, and more.

A guide to using this API, called E-Utilities, to access citation data for medical journal literature in PubMed and other NCBI databases, including the National Library of Medicine Catalog, MeSH, Gene, and PMC (PubMed Central).

360 years of United States caselaw available via API and bulk downloads. (Downloads for some jurisdictions are only available to scholars who sign a research agreement.)

Textual datasets and bibliometric data from a variety of Canadian sources including hundreds of Érudit journals, archives from a variety of national and provincial collections from the 1800s to present, parliamentary debates, and more.

Opinions, docket files, and more from 420 courts.

Bulk data downloads of major US Government publications including Congressional Bills, Commerce Business Daily, Federal Register, Public Papers of the Presidents of the United States, Supreme Court Decisions 1937-1975 (FLITE) and more.

Use this REST API to access full-text and metadata from FRASER, a digital library of U.S. economic, financial, and banking history—particularly the history of the Federal Reserve System.

A variety of data repositories from the Library of Congress related to business, demographics, news, images, science, government and more.

The ProPublica Congress API has up-to-date legislative data from the House of Representatives, the Senate and the Library of Congress, including details about members, votes, bills, nominations and more. ProPublica also offers free APIs and data on Campaign Finance, Trump Appointees, Political Ad Buys, and more.

Widely used free linguistic corpora created by Mark Davies, Professor of Linguistics at Brigham Young University. Now hosted at english-corpora.org. Sign up for a free academic account to work with many of these corpora (listed below).

The corpus contains more than 560 million words of text (20 million words each year 1990-2017) and it is equally divided among spoken, fiction, popular magazines, newspapers, and academic texts.

COHA contains more than 400 million words of text from the 1810s-2000s.

CoRD provides links to and descriptions of a large number of corpora, subcorpora and databases. (University of Helsinki)

The LDC collects language data from both written texts and transcriptions of speech, in various languages, to support corpus linguistics. The Library subscription, and access to LDC datasets, begins in 2022 though some older datasets are also available.

200 million words of data in more than 25,000 movies from the 1930s to present. All movies are tied in to their IMDB entry.

7.1 billion words of data (and growing) from web-based newspapers and magazines from 2010 to the present.

15 million words of American English automatically annotated for logical structure, word and sentence boundaries, part of speech (multiple tag sets), shallow parse (noun and verb chunks), and named entities.

The Scottish Corpora project has created large electronic corpora (over 4.5 million words) of written and spoken texts for the languages of Scotland. See also the Helsinki Corpus of Older Scots (1450-1700) and the Corpus of Modern Scottish Writing (1700-1945).

325 million words of data from 75,000 TV episodes from the 1950s to the present. All episodes are tied in to their IMDB entry.

14 billion words from 22 million web pages and 95k websites.

A list of topic-centric high quality public data sources. Collected from blogs, answers, and user responses.

The collected posts of 19,320 bloggers gathered from blogger.com in August 2004. The corpus incorporates a total of 681,288 posts.

The Common Crawl corpus contains petabytes of data collected from the web since 2008. It contains raw web page data, extracted metadata and text extractions.

This dataset includes two tables: data on the interest categories Facebook shows to users and the ad groups its shows to advertisers (2016).

How to download files from archive.org in an automated way using wget.

A directory of sites archiving various social media posts from members of the Obama Administration.

This database, updated daily, contains ads that ran on Facebook and were submitted by thousands of ProPublica users from around the world (via browser extensions).

Access data from posts, threads, comments, users and more from reddit and subreddits. Historical Reddit data has been collected at http://files.pushshift.io/reddit/ as monthly CSV downloads.

As a service to the Machine Learning, Data Mining, and Social Sciences communities, the Social Computing data repository currently hosts datasets from a collection of many different social media sites.

A centralized repository of researcher social media data via ICPSR. Open to submissions of social media research datasets.

The SNAP library collects data on large social and information networks since 2004.

Directory of open-access tweet datasets on DocNow, available for research use. To convert Tweet IDs to JSON files of full-text tweets and metadata, use DocNow's Hydrator app. See also: UNLV's Twitter Data Tutorial Series.

35,108 Tweets curated from seven different domains of Twitter corpus that have been hand-annotated for 10 categories of moral sentiment.

14 billion words from 22 million web pages and 95k websites.

Monthly database backups of all Wikimedia wikis in various formats.

Check out the library's guide to subscription data archives and resources.