XKEYSCORE OPERATIONS – LIFTING THE HOOD
The XKeyscore’s reach across the global web traffic is so intense and well entrenched that there is a fair chance that web users’ day to day activities like emails, social media accesses, online purchases etc., would be going through the cross hairs of this data collecting behemoth’s targeting system.
With 700 servers scattered globally, almost everything gets scanned and processed. To streamline the process of data collection, intelligent software automatically categorizes data based on its type – emails are tied with their provides, for instance Gmail or Hotmail – its contents etc.
This massive database is, in fact, a global collection of networked data collecting servers running on Linux software, which, akin to a google search, is accessed for required data through a simple search engine through web browsers.
XKeycore servers are scattered around 150 sites globally. Each site comprises a cluster of servers, the number of which varies depending on the amount of information being acquired at that site. Logically enough, sites processing greater traffic and storage require more servers for sifting and parsing collected data. An ingenious system has been developed to augment processing power and storage by simply adding more servers to a cluster. As per records, there are roughly 700 servers comprising the XKeyscore data collection section. To give an idea of the volume of traffic being processed, some sites have to filter over 20 tera-bytes of data per day, an equivalent of around 13000 movies or 5.7 million songs.
The XKeyscore powered by Linux software that is usually deployed on Red Hat servers. Data collected is stored in MySQL databases utilizing the Apache web server. Since data is distributed on numerous servers, file handling is managed in a cluster by NFS distributed file system and the autofs service. The cron scheduling service is used to handle scheduled tasks. XKeyscore’s servers are maintained by tools such as vim and rsync along with a thoroughly integrated command line tool by its system administrators. XKeyscore is accessed by analysts simply through an internet connection with a web browser other other than Internet Explorer. The login to the search engine is through login credentials – ID and a password. Alternatively, public key authentication can also be used.
The data stored in MySQL databases harvested from collection sites is extracted by the XKeyscore search engine using a federated query system – query is sent from the main XKeyscore server and relayed over the internet to all the field sites, effectively running the query on all servers simultaneously. The servers then extract relevant data and send it back to the search indenter via the central server. All this managed through various techniques, of which, tagging meta data is the most prominent one that makes this highly intricate task manageable and effective. The system is powerful enough to track activities of targets of interests based on location, nationality and websites viewed. this information can then also be corelated to reveal foreigners in a certain country, visiting websites in a specific language. This is just the tip of the ice berg, the power of XKeyscore extends well beyond these examples.