-
Notifications
You must be signed in to change notification settings - Fork 37
Handling huge document sets
Overview is designed to support document sets that contain tens of thousands of documents or more. (Our eventual goal is 10 million.) If you'd like to try to get the most out of your own installation of Overview, here are some tips.
Before reading these tips, try handling a huge document set. Maybe it will work! If it doesn't, read on....
If you see an OutOfMemoryError, your document set is too large for the worker process. Try increasing the -Xmx parameter of your worker instance.
There is a specific case of this worth mentioning: 32-bit Java. Unfortunately, 32-bit versions of Java don't allow you to increase -Xmx beyond around 1.5 gigabytes. So make sure you're running a 64-bit version of Java: type java -version on the command-line and scan for the key words, "64-bit". If it doesn't say "64-bit" it isn't 64-bit; uninstall Java and install a 64-bit version from the JDK download page.
(Note: 64-bit Java won't run on 32-bit Windows machines. Also, Java's automatic installers tend to default to a 32-bit version on many machines, which is why we linked to the JDK download page directly. See (the reasons)[http://java.com/en/download/faq/java_win64bit.xml] if you're curious.)
CSV upload is the quickest to upload and parse, so try it if you're frustrated by other methods.
At Overview, we love working with huge document sets. Ask the (overview-users)[https://groups.google.com/forum/#!forum/overview-users] group for help and you'll probably get some good advice.