Managing files on ScienceData with Python

First, as you've probably noticed, the files you browse and manage via the notebook file manager are in fact your files on ScienceData.

The ScienceData Notebook Service builds on the ScienceData Pod Service: Each notebook server runs in a pod owned by you. A pod has its own local storage, hosting the system files plus perhaps home directories of users etc. This storage is volatile, i.e. when the pod is shut down it is destroyed with the pod.

The local storage is fast, but should be used only for temporary files. To keep files permanently, the typical procedure is to write them locally and then copy them to ScienceData.

For this reason, we've tried to make accessing your ScienceData files from your pods as easy as possible. Read on to learn the basics.

First we need to import the libraries we'll be using:

requests is the library we'll be using to access files on ScienceData via HTTP.

os is the basic library we'll be using to run commands in the pod we're running in.

Notice The hostname we'll be using, sciencedata, is known only on the private subnet to which your pods are connected. It is mapped to the private IP address of your ScienceData home server. You can access it w/o username/password. This is convenient, but has the downside that authentication is slow. If you run out of patience with this, you can provide username/password with requests, but we do not recommend hard-coding your password in your notebooks. Instead, use the SDDK library, which will prompt you for your password.
In [2]:
import os
import requests

Let's try and write some text to a local file.

In [3]:
f = open("my_file.txt", mode='w', encoding='utf-8')

f.write("this is a test");

f.close()

Check that the text got written.

In [26]:
f = open("my_file.txt", mode='r', encoding='UTF-8')

text = f.read()

f.close()

text
Out[26]:
'this is a test'

Copy the file to ScienceData. If you don't have the directory tmp, you can head over to sciencedata.dk and create it. Or use another directory.

In [28]:
f = open("my_file.txt", mode='rb')

with f as file:

    requests.put("https://sciencedata/files/tmp/my_file.txt", file)

f.close()

Check that the file got written.

In [29]:
r = requests.get("https://sciencedata/files/tmp/my_file.txt")

r.text
Out[29]:
'this is a test'

Get a binary file and write it to the local disk.

In [3]:
r = requests.get("https://sciencedata.dk/public/9e48cffd2f1fa3fada752877f490d5d1/cover.png")
with open('cover.png','wb') as f:
    f.write(r.content)

Another way to access files, both locally and on ScienceData, is to call command-line tools in a shell on the host pod/container.

In [30]:
os.system("curl --insecure https://sciencedata/files/tmp/my_file.txt");

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:13 --:--:--     0
this is a test
100    14  100    14    0     0      1      0  0:00:14  0:00:13  0:00:01     3
In [ ]: