Mobile Cloud And FUSE

Its all started when I wanted to migrate to another mobile company. The market is very competitive and the prices are all about the same. One of the company offered a cloud backup service. Further investigation shows that in order to compete with other backup services the connection to this backup cloud does not comes out from your data plan.  Now they got my attention *ultimate mobile traffic anyone?*. I said to myself I will take this plan only to "play" with their cloud.


I started to "play" around the cloud and really soon realized how primitive their software is: A crappy mobile app that does not really do anything beside restore and backup, and a crappy PC software which looks like windows 95.  I hope for some virtual drive capability like Google Drive but on reality not even close. I decided that my first move is to make virtual drive over the cloud API. The cloud api is really basic http web api, something like upload_file.aspx, download_file.aspx and list_files.aspx. I started to implement a filesystem over this web api. I used FUSE which is basically kernel code that let the user-mode write filesystems. In order to simplify things there is this FUSE python module called fusepy which let you write filesystems in python! here is an example for memory based filesystem:

	#!/usr/bin/env python
	from collections import defaultdict
	from errno import ENOENT
	from stat import S_IFDIR, S_IFLNK, S_IFREG
	from sys import argv, exit
	from time import time
	from fuse import FUSE, FuseOSError, Operations, LoggingMixIn
	class Memory(LoggingMixIn, Operations):
	"""Example memory filesystem. Supports only one level of files."""
	def __init__(self):
	self.files = {}
	self.data = defaultdict(str)
	self.fd = 0
	now = time()
	self.files['/'] = dict(st_mode=(S_IFDIR | 0755), st_ctime=now,
	st_mtime=now, st_atime=now, st_nlink=2)
	def chmod(self, path, mode):
	self.files[path]['st_mode'] &= 0770000
	self.files[path]['st_mode'] |= mode
	return 0
	def chown(self, path, uid, gid):
	self.files[path]['st_uid'] = uid
	self.files[path]['st_gid'] = gid
	def create(self, path, mode):
	self.files[path] = dict(st_mode=(S_IFREG | mode), st_nlink=1,
	st_size=0, st_ctime=time(), st_mtime=time(), st_atime=time())
	self.fd += 1
	return self.fd
	def getattr(self, path, fh=None):
	if path not in self.files:
	raise FuseOSError(ENOENT)
	st = self.files[path]
	return st
	def getxattr(self, path, name, position=0):
	attrs = self.files[path].get('attrs', {})
	try:
	return attrs[name]
	except KeyError:
	return '' # Should return ENOATTR
	def listxattr(self, path):
	attrs = self.files[path].get('attrs', {})
	return attrs.keys()
	def mkdir(self, path, mode):
	self.files[path] = dict(st_mode=(S_IFDIR | mode), st_nlink=2,
	st_size=0, st_ctime=time(), st_mtime=time(), st_atime=time())
	self.files['/']['st_nlink'] += 1
	def open(self, path, flags):
	self.fd += 1
	return self.fd
	def read(self, path, size, offset, fh):
	return self.data[path][offset:offset + size]
	def readdir(self, path, fh):
	return ['.', '..'] + [x[1:] for x in self.files if x != '/']
	def readlink(self, path):
	return self.data[path]
	def removexattr(self, path, name):
	attrs = self.files[path].get('attrs', {})
	try:
	del attrs[name]
	except KeyError:
	pass # Should return ENOATTR
	def rename(self, old, new):
	self.files[new] = self.files.pop(old)
	def rmdir(self, path):
	self.files.pop(path)
	self.files['/']['st_nlink'] -= 1
	def setxattr(self, path, name, value, options, position=0):
	# Ignore options
	attrs = self.files[path].setdefault('attrs', {})
	attrs[name] = value
	def statfs(self, path):
	return dict(f_bsize=512, f_blocks=4096, f_bavail=2048)
	def symlink(self, target, source):
	self.files[target] = dict(st_mode=(S_IFLNK | 0777), st_nlink=1,
	st_size=len(source))
	self.data[target] = source
	def truncate(self, path, length, fh=None):
	self.data[path] = self.data[path][:length]
	self.files[path]['st_size'] = length
	def unlink(self, path):
	self.files.pop(path)
	def utimens(self, path, times=None):
	now = time()
	atime, mtime = times if times else (now, now)
	self.files[path]['st_atime'] = atime
	self.files[path]['st_mtime'] = mtime
	def write(self, path, data, offset, fh):
	self.data[path] = self.data[path][:offset] + data
	self.files[path]['st_size'] = len(self.data[path])
	return len(data)
	if __name__ == "__main__":
	if len(argv) != 2:
	print 'usage: %s <mountpoint>' % argv[0]
	exit(1)
	fuse = FUSE(Memory(), argv[1], foreground=True)

view raw gistfile1.py hosted with ❤ by GitHub

As you can see, you can implement the open, write, read, etc in python and basically you got working filesystem.. pretty cool I say. Using python requests module and fusepy I implemented all the filesystem functions I needed over the web api. One of the problems I encountered is that the basic web api server does not support "Range" http header, meaning I cannot seek, and a filesystem without seek cant really do anything. Then I discovered they have a share feature, when you press share you get a link to share your file. This link is on another IIS server which supports the range header, so I am able to seek. In order to make this work in the filesystem I programmed every open request to share the file, and save the share token in some array at the memory and cache it back in the next open. Also all the content on the cloud is encrypted to prevent copyright abuse requests. The filesystem automatically decrypt the content, and the softwares accessing the filesystem does not aware of that, as they read files from the filesystem they get un-encrypted content.
For now, this filesystem is read only because I didn't think of a way to seek on http upload form.

Because the filesystem is read-only, on the VPS\dedicated\PC I got a python script which can get a local file as argument and upload it to the cloud. At this point I can tell my server to torrent a movie for example, upload it to the cloud and I can watch it on the bus over LTE - on vlc - which getting the file from my cloud encrypted filesystem - without draining my data plan.

Next thing is to get Tunnel Over File, I am thinking with some sort of virtual TAP\TUN device constantly sampling the cloud files for received packets in the incoming range. The outgoing range it will upload each packet to the cloud for the other side to sample, but I don't think I will do it.. too much work and too much abuse of the service.

Thats it for this post..