Extracting patterns and intelligence from large datasets, often without subjects' knowledge
Data mining is the process of analyzing large datasets to discover patterns, correlations, and useful information. In commercial contexts, data mining drives targeted advertising, credit scoring, price discrimination, and consumer profiling. In government and intelligence contexts, data mining enables mass surveillance, predictive policing, social network analysis, and threat identification.
The NSA's surveillance programs revealed by Snowden were fundamentally data mining operations — collecting vast quantities of communications data and using algorithms to identify patterns of interest. The Total Information Awareness (TIA) program, proposed by DARPA in 2002 under the direction of convicted Iran-Contra figure John Poindexter, envisioned mining financial transactions, travel records, communications, and other data sources to predict terrorist activity. Congress nominally defunded TIA in 2003 after public outcry, but investigative reporting revealed that its component programs continued under different names within the NSA.
The commercial data mining industry has created an infrastructure that governments can access through legal demands, partnerships, or purchases. Data brokers like Acxiom, LexisNexis, and Palantir compile detailed profiles on hundreds of millions of individuals. The FBI, ICE, and other agencies have purchased commercial data to circumvent warrant requirements — if the data is "commercially available," agencies argue they don't need a warrant to access it.
