Skip to content
This repository has been archived by the owner on Apr 14, 2020. It is now read-only.

Migrating SCV from Python to Go #59

Open
proteneer opened this issue Dec 8, 2014 · 1 comment
Open

Migrating SCV from Python to Go #59

proteneer opened this issue Dec 8, 2014 · 1 comment

Comments

@proteneer
Copy link

Sorry for the delay.

I've finished the Go port of SCV and have started the migration by merging into this repo. There's been about 100+ commits coming in all at once. The migration should take about 2-3 days to complete as I upgrade the CC's on AWS. I need to make some additional modifications on the CC as well. @mpiercy we might want to do this together.

Key features:

  1. No longer dependent on redis/LUA to hack together custom queries. This was probably the biggest reason why I made the switch. You have to understand python's GIL at one point became the bane of my existence, and to allow multi-processing to work, you had to coordinate through an external service (such as redis) and fork your processes. As time went on, I needed more than the basic pipelining operations available in redis, and had to basically hack using LUA (think CUDA's equivalent of JIT).
  2. Insanely parallelizable. The single-nature nature of redis meant that all memory-related queries were serialized. Before, this effectively meant that only one stream could be modified at any given time. The new ReadWrite mutex based Go code means that we had N-way parallelism, where N is the number of streams. This is an insane speed-up. It's really really fast.
  3. Go server is effectively stateless, meaning we don't need to rely on the redis cache to persist across sessions. The server can self recover directly from the MongoDB and what ever binary data exists on disk.
  4. Globally accessible stream information inside the DB. Originally, because of sharding and perhaps a bad design decision on my part, stream information was only accessible on SCVs. If an SCV goes off-line the state of the stream is lost. In this case, stream presistence is enabled through MongoDB, though in a slightly dirty state perhaps.
  5. Go binaries are statically built, and will be made available soon. No more cmake, scons, make, pip or any of that garbage (for the SCV)! Go binaries are compiled to bytecode directly, cross-platform compilation is insanely easy.

Why Go and not java? Because of goroutines. Each HTTP request essentially exists in its own goroutine, and the overhead of goroutines in Go is a very very tiny fraction of that of an OS thread you'd find in C++/Java. Python's tornado is different, as it has its own built-in async event poll.

In short, Go is amazing.

@proteneer
Copy link
Author

I should note that while the SCV is done, there is still some integration work to be done. I'm working on fixing the integration tests and siegetank tests.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant