Skip to content

Latest commit

 

History

History
297 lines (223 loc) · 11.3 KB

CHANGES.md

File metadata and controls

297 lines (223 loc) · 11.3 KB

Change Log

Unreleased

Nothing notable yet.

0.10.0 - 2018-07-15

Added

  • #242: Flintrock is now available on Homebrew: 
    brew install flintrock
    This is a community-supported distribution.

Changed

  • #224: Fixed a problem with some Flintrock config combinations related to Hadoop.
  • #232: When you destroy a cluster, Flintrock now waits until the instances are completely terminated before returning.
  • #234: Flintrock now tries more times by default to connect via SSH, which should provide more launch stability in certain environments.
  • #246: Fixed some bugs with flintrock describe that are exposed when a cluster is transitioning states (e.g. from running to terminated).
  • #249: Flintrock now downloads both Spark and Hadoop from Apache mirrors by default. This is a significant change. You can read the background on what prompted this change in #238.
  • #254: Flintrock no longer configures hadoop-aws automatically due to version incompatibilities that are difficult to resolve automatically. Instead, the README now provides additional guidance on using s3a://.
  • #259: Flintrock now correctly ignores tiny devices that show up on some instance types, like the M5 series on EC2. This fixes the problems Flintrock had getting HDFS to work on those instance types.

0.9.0 - 2017-08-06

Added

  • #178: You can now see additional output during launch and other operations with the new --debug option.
  • #185: Added a new mount point under /media/tmp that can be used when /tmp is not big enough.
  • #186: You can now tag your clusters with arbitrary tags on launch using the new --ec2-tag option. (Remember: As with all options, you can also set this via flintrock configure.)
  • #191: You can now specify the size of the root EBS volume with the new --ec2-min-root-ebs-size-gb option.
  • #181: You can now set the number of executors per worker with --spark-executor-instances.

Changed

  • #195: After launching a new cluster, Flintrock now shows the master address and login command.
  • #196, #197: Fixed some bugs that were preventing Flintrock from launching Spark clusters at a specific commit.
  • #204: Flintrock now automatically retries starting the Spark and HDFS masters if it encounters common issues with bringing the cluster up. This greatly improves launch and restart reliability.
  • #208: Flintrock now provides a hint with possible causes for certain SSH errors.

0.8.0 - 2017-02-11

Added

  • #180: Accessing data on S3 from your Flintrock cluster is now much easier! Just configure Flintrock to use Hadoop 2.7+ (which is the default) and an appropriate IAM role, and you'll be able to access paths on S3 using the new s3a:// prefix. Check the README for more information.
  • #176, #187: Flintrock now supports users with non-standard home directories.

Changed

  • #168: Flintrock now does a better job of cleaning up after interrupted operations.
  • #179, #184: Flintrock can now clean up malformed Flintrock clusters.
  • 6b426ae: We fixed an issue affecting some users of Flintrock's standalone package that caused Flintrock to intermittently throw ImportErrors.

0.7.0 - 2016-11-15

Added

  • #146: Flintrock now ensures that launched clusters have Java 8 or higher installed.
  • #149: You can now specify an EC2 user data script to use on launch with the new --ec2-user-data option.

Changed

  • #154, #155, #156: Flintrock now provides friendly error messages when it encounters common configuration or setup problems.

0.6.0 - 2016-08-28

Added

  • #115: Flintrock can now resize existing clusters with the new add-slaves and remove-slaves commands.

Changed

  • #115: If you lost your master somehow, Flintrock can now still destroy the cluster.
  • #115: You can no longer launch clusters with 0 slaves. The implementation was broken. We may fix and add this capability back in the future.

0.5.0 - 2016-07-20

Added

  • #118: You can now specify --hdfs-download-source (or the equivalent in your config file) to tell Flintrock to download Hadoop from a specific URL when launching your cluster.
  • #125: You can now specify --spark-download-source (or the equivalent in your config file) to tell Flintrock to download Spark from a specific URL when launching your cluster.
  • #112: You can now specify --ec2-security-group to associate additional security groups with your cluster on launch.

Changed

  • #103, #114: Flintrock now opens port 6066 and 7077 so local clients like Apache Zeppelin can connect directly to the Spark master on the cluster.
  • #122: Flintrock now automatically adds executables like spark-submit, pyspark, and hdfs to the default PATH, so they're available to call right when you login to the cluster.

0.4.0 - 2016-03-27

Added

  • #98, #99: You can now specify latest for --spark-git-commit and Flintrock will automatically build Spark on your cluster at the latest commit. This feature is only available for Spark repos hosted on GitHub.
  • #94: Flintrock now supports launching clusters into non-default VPCs.

Changed

  • #86: Flintrock now correctly catches when spot requests fail and bubbles up an appropriate error message.
  • #93, #97: Fixed the ability to build Spark from git. (It was broken for recent commits.)
  • #96, #100: Flintrock launches should now work correctly whether the default Python on the cluster is Python 2.7 or Python 3.4+.

0.3.0 - 2016-02-14

Changed

  • eca59fc, 3cf6ee6: Tweaked a few things so that Flintrock can launch 200+ node clusters without hitting certain limits.

0.2.0 - 2016-02-07

Added

  • b00fd12: Added --assume-yes option to the launch command. Use --assume-yes to tell Flintrock to automatically destroy the cluster if there are problems during launch.

Changed

  • #69: Automatically retry Hadoop download from flaky Apache mirrors.
  • 0df7004: Delete unneeded security group after a cluster is destroyed.
  • 244f734: Default HDFS not to install. Going forward, Spark will be the only service that Flintrock installs by default. Defaults can easily be changed via Flintrock's config file.
  • de33412: Flintrock installs services, not modules. The terminology has been updated accordingly throughout the code and docs. Update your config file to use services instead of modules. Warning: Flintrock will have problems managing existing clusters that were launched with versions of Flintrock from before this change.
  • #73: Major refactoring of Flintrock internals.
  • #74: Flintrock now catches common configuration problems upfront and provides simple error messages, instead of barfing out errors from EC2 or launching broken clusters.
  • bf766ba: Fixed a bug in how Flintrock polls SSH availability from Linux. Cluster launches now work from Linux as intended.

0.1.0 - 2015-12-11

  • Initial release.