-
Notifications
You must be signed in to change notification settings - Fork 37
Handling huge document sets
Overview is designed to support document sets that contain tens of thousands of documents or more. (Our eventual goal is 10 million.) If you want to handle a large document set, here are some hints.
Before reading these tips, try simply uploading your huge document set. Maybe it will work! If it doesn't, read on....
If you see an OutOfMemoryError, your document set is too large for the worker process. Try increasing the -Xmx parameter of your worker instance.
Windows users, beware: avoid 32-bit Java if you want to upload tens of thousands of documents. Unfortunately, 32-bit versions of Java don't allow you to increase -Xmx beyond around 1.5 gigabytes. Make sure you're running a 64-bit version of Java: type java -version on the command-line and scan for the key words, "64-bit". If it doesn't say "64-bit" it isn't 64-bit; uninstall Java and install a 64-bit version from the JDK download page.
(Note: 64-bit Java won't run on 32-bit machines. Also, avoid Java's automatic installers. They tend to default to a 32-bit version on many 64-bit machines, which is why we linked to the JDK download page directly. See (the reasons)[http://java.com/en/download/faq/java_win64bit.xml] if you're curious.)
If you set -Xmx too high, Overview may munch away at all your system's memory until the kernel kills it. When that happens, you'll see nothing but a graceless Process killed message on your console. Overview should behave equivalently after either Process killed or OutOfMemoryError, so pick your poison.
CSV upload is the quickest to upload and parse, so try it if you're frustrated by other methods.
At Overview, we love working with huge document sets. Ask the (overview-users)[https://groups.google.com/forum/#!forum/overview-users] group for help and you'll probably get some good advice.