Skip to content

Latest commit

 

History

History
249 lines (131 loc) · 15.9 KB

rclone-colab.md

File metadata and controls

249 lines (131 loc) · 15.9 KB

Get started

Open in app

Homepage

Towards Data Science

Sign in

Get started

Homepage

Follow

602K Followers

·

Editors' PicksFeaturesDeep DivesGrowContribute

About

Get started

Open in app

Homepage

Why You Should Try Rclone with Google Drive and Colab

Wouldn’t it be nice if you never lost your trained ML models on the Google Colab? Rclone to the rescue.

Steven Smiley

Steven Smiley

Jun 17·4 min read

Image by author.

Do you ever start training a machine learning model on the Google Colab and the runtime disconnects before you can save your results? It use to happen to me and I would get frustrated. I would leave my model training to come back a few hours later and find it finished without saving!

Image by author.

So what can you do about this? Use Rclone¹, Google Drive², and Google Colab³ together with the following 3 major steps:

1) Install Rclone.

2) Configure Rclone.

3) Export with Rclone.

Step 1. Install Rclone.

Do this in the very first cell of your Google Colab Jupyter Notebook.

<span id="3c3b" class="hr mn lk gu mo b dn mp mq s mr">! curl https://rclone.org/install.sh | sudo bash</span>

Step 2. Configure Rclone.

Configure the newly installed Rclone.

<span id="ff6d" class="hr mn lk gu mo b dn mp mq s mr">!rclone config</span>

Step 2.1. Make a new remote

Step 2.2. Name it “remote”

Step 2.3. Enter “15” for the Google Drive

Step 2.4. Press Enter for default on client_id

Step 2.5. Press Enter for default on client_secret

Step 2.6. Enter “1” for Full access all files on your Google Drive.

Step 2.7. Press Enter for default on root_folder_id.

Step 2.8. Press Enter for default on service_account_file.

Step 2.9. Enter “n” for default on advanced config.

Step 2.10. Enter “n” for non-default on Remote config.

Go to the link it generates for you.

Copy the code from your Google Drive Sign in for Rclone access.

Paste it where it tells you to Enter verification code

Step 2.11. Enter “n” for default on Shared Drive.

Step 2.12. Enter “y” for default on remote settings.

Step 2.13. Enter “q” for Quit config.

Step 3. Export with Rclone.

Add an export after your model finishes training.

Copy your current “/content/” directory in the Google Colab to wherever you want on your Google Drive through the Rclone remote path we just established.

<span id="ebce" class="hr mn lk gu mo b dn mp mq s mr">!rclone copy "/content/"  remote:"/YOUR_PATH_TO_GDRIVE_DESIRED_LOCATION/"</span>

Summary

So after following those 3 major steps outlined above, you should be able to back-up whatever models you are training with Google Colab (assuming they finish).

Here is an example Jupyter Notebook that implements theses steps from start to finish.

You can see for that example, my Google Drive stored the trained models as such under content:

Image by author.

You can find more examples in this Github repository I made for various TensorFlow machine learning problem notebooks.

I hope you found this article helpful! Thank you for reading and hit the like if so. Follow me on here for more machine learning based content coming soon!

You can also follow me on LinkedIn: https://www.linkedin.com/in/stevensmiley1989/

References

  1. Rclone.[ https://rclone.org/]
  2. Google Drive. [ https://g.co/kgs/qu7aAY]
  3. Google Colab. [ https://colab.research.google.com/]
  4. TensorFlow. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis,Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Rafal Jozefowicz, Yangqing Jia,Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Mike Schuster,Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Jonathon Shlens,Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker,Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas,Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke,Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available fromtensorflow.org.

Steven Smiley

Writing about Data Science, CV, DL, ML, AI, Python https://www.linkedin.com/in/stevensmiley1989/

Follow

45

45

45

More from Towards Data Science

Follow

Your home for data science. A Medium publication sharing concepts, ideas and codes.

Read more from Towards Data Science

More From Medium

Automatic Video Editing using Python

Dmytro Nikolaiev (Dimid)in Towards Data Science

Big Data will First Slow, Not Accelerate, Discovery

Eric Luellen

the least among high-income developed nations at 4.3%

Allsportslivenow

What is true impact?

Arun Ramanujapuramin logistimo

Visualization and Multiple imputation

Rabi Kumar Singh

Exploring Greater Sydney suburbs

Tariq Munirin Towards Data Science

A Brief insight on Data Pre-processing and Visualization for Machine Learning Model

Suhana Anjum

Stack Index 📇

Vaishnavi Vin featurepreneur