README

88888888888                               d88P  .d8888b.                888          
    888                                  d88P  d88P  Y88b               888          
    888                                 d88P   888    888               888          
    888  888  888 88888b.   .d88b.     d88P    888         .d88b.   .d88888  .d88b.  
    888  888  888 888 "88b d8P  Y8b   d88P     888        d88""88b d88" 888 d8P  Y8b 
    888  888  888 888  888 88888888  d88P      888    888 888  888 888  888 88888888 
    888  Y88b 888 888 d88P Y8b.     d88P       Y88b  d88P Y88..88P Y88b 888 Y8b.     
    888   "Y88888 88888P"   "Y8888 d88P         "Y8888P"   "Y88P"   "Y88888  "Y8888  
              888 888                                                                
         Y8b d88P 888                                                                
          "Y88P"  888                                                                
    
    Wikileaks CableGate Processing
    andrew@typeslashcode.com
    v0.0.0.0.0.2
    
    Prerequisites:
      HTTrack (http://www.httrack.com/httrack-3.43-9C.tar.gz)
        *to update mirror of cablegate site
      MongoDB (www.mongodb.org)
        *database to store parsed cables
        *ouputs json dump
    
    Functionality:
      Will update Cablegate Web Mirror, and pull all existing cables into a 
        MongoDB Collection where their 'Reference ID' is their '_id', and they
        contain the following properties:
          'reference_id','date_time','classification','origin','header','body'
    
    Usage:
      Make sure that mongod is running. The software is configured to access
        Mongod at it's default location, so change that if necessary.
      Confirm that httrack and mongoexport are accesable in your PATH.
      run 'python process.py'
      after that
      run 'mongoexport -d wikileaks -c cables -o dump/cables.json'
      
    Notes:
      The Tornado app that is there right now serves no function. To come..?