SkyDocs and The Graph - Final Grant Report

At the end of March 2021, I heard I was selected for a grant from The Graph to work on SkyDocs. This report describes all the things I've worked on and includes link to more in depth information.
Check out SkyDocs: https://skydocs.hns.siasky.net

What is SkyDocs?

SkyDocs is a decentralized Google Docs alternative. It uses Sia for storage. Sia is a decentralized cloud storage platform. They have created Skynet, which is a Content Delivery Network for the Sia network. They provide easy APIs you can use in your dApp.

SkyDocs was originally created for a Sia Skynet Hackathon. It was built in a few days. This grant means I can take it to the next level!

Grant Goals

The grant provided me with enough time to work on several things. At the start, these goals were defined:

The main goal of the grant was to allow the SkyDocs app to share private and public documents. We are going to use The Graph to enable this functionality. You will be able to share documents using a subgraph and see which documents are shared with you.

New UI

The first version of the app was created for a hackathon. During the hackathon, there were enough technical challenges to get the app working on Sia Skynet, so the UI didn't get much attention. Now there's time to do that!

MetaMask integration

The Graph indexes data from blockchains like Ethereum. If we want to let SkyDocs write data about shared documents to The Graph, we need to write this to the blockchain so The Graph can index it. Using MetaMask, we can interact with smart contracts on the Ethereum blockchain from the browser. So, we need to integrate MetaMask and SkyDocs.

Subgraph

Data can be indexed and queried from a subgraph. Maybe there is an existing subgraph we can use that supports our sharing setup, or I will need to create a new subgraph for that.

Developer tooling

This is all new technology! I'm probably the first person using The Graph from a C# Blazor WebAssembly app. So why not make my tooling available to other developers? Of course I will! Everything I create will be open source and this grant will result in multiple libraries other developers can use.

Development and results

SkyDocs

The most visible difference in SkyDocs is of course the UI. These screenshots show the differences.

Old login New login

But it's not the only thing. I've also given the development workflow of SkyDocs a big update. SkyDocs is written in C# with Blazor and compiles to WebAssembly. This is pretty new technology and not everything is working out of the box.
First, I made sure I could properly debug the application. If I'm working on the app full time, I can't live without proper debugging. I wrote this blog post about Debugging Blazor WebAssembly.

I've also adopted GitHub Actions to do all the building and continuous deployment. I used the deploy-to-skynet-action GitHub Action to automatically deploy every Pull Request to Skynet. This makes testing new changes very easy.
Every commit on the master branch gets automatically deployed as the latest version on https://skydocs.hns.siasky.net

With all these auto deployments, it's also useful to see which version of the app you're using. I blogged about how to show your app version using GitHub Actions and Blazor.

Finally, I noticed that my app got bigger and bigger in file size. To give users the best experience, the download size should be small. I blogged about reducing the download size of Blazor WebAssembly apps.

To be able to share documents, I had to refactor SkyDocs and rewrite the logic of how documents are stored.
When you login, you create a private and public key. In the first version of SkyDocs, all documents were written to storage using that private key, and you can update the document if you know the private key. Documents were not encrypted.

We want to add encryption, read-only sharing and editor-sharing.

Every document gets its own encryption key. If you know the document id and the encryption key, you can read the document. This enables read-only sharing. I've modified the URL structure so that when you're editing a document, you can share the URL with anybody to share a read-only view of the document.

Example: https://skydocs.hns.siasky.net/?id=4f463818-19b5-44c3-abe6-d4532a7db768&pub=67CB3FFB1F7784344FF5B50E1313C7E06A9AD3076960C0C534FC620C41633370&c=8de0a347-e8be-4725-910a-b69437ee519e

Another change is that every document gets its own public/private keypair. If you know the private key, you can update the document.

So now every document has 3 important properties:

Document id: we know there is a document here
Encryption key: we can read the document
Private key: we can write to this document

A document sharing popup dialog was added to easily share documents using a URL.

Share Dialog

MetaMask and Blazor

SkyDocs is a decentralized app. There is no central authority that checks your username and password. If you login with SkyDocs, ANY username/password combination just works. How is that possible?
Your username and password are used as seed to generate a public and private key. Those keys are used to read and write data from Skynet.

MetaMask is a browser plugin, it allows you to send and receive tokens on the Ethereum blockchain and interact with smart contracts. It also has a JavaScript API developers can use to interact with it.
MetaMask controls your public and private key for the Ethereum blockchain. We want to use these keys to identify a user. But MetaMask will never give your private key to a third-party app.
There is however the option to sign text with your private key in MetaMask. Only the owner of a private key can sign, this results in a hash. The hash can be made public and verified with the public key. So, if a user provides a valid signed string, we can identity the user and login with SkyDocs.

I've blogged about this setup here: Logging in with MetaMask in a Blazor WebAssembly app

This work also resulted in a library other developers can use to add login with MetaMask to their Blazor WebAssembly apps. The library is open source and available on GitHub (MetaMask.Blazor).

The Graph

Everything is SkyDocs was now ready for the icing on the cake. Integrating with The Graph! When you login with MetaMask, this unlocks an additional document share option. You can share the document with another Ethereum address.

All the properties that are needed to be able to open a shared document are stored as a JSON file on Sia Skynet. The URL of this file (sia://hash) is then send to a smart contract I developed. This smart contract emits an event which is then indexed and made available on a subgraph from The Graph.

In this blog post you can read more about my experience creating a subgraph.

SkyDocs queries this subgraph and shows you which documents are shared with you.

I had to create a smart contract, deploy it to the Ethereum Kovan testnet and write tooling to query The Graph from my C# app.

This has resulted in a generic solution that I've made open source. It's called The ShareIt Network.

The ShareIt Network

More info about this architecture can be found here: https://github.com/michielpost/SkyDocs/tree/master/ShareArchitecture

To easily query The Graph from C# apps, I've created a C# client library. It's open source and can be found on GitHub (TheGraph.Client).

I was happy to see this library already helped another developer who was learning about The Graph https://github.com/michielpost/TheGraph.Client/issues/1

What could be better?

Subgraph development

The data from the smart contract goes through a mapping file which results in the data that will be indexed. In this mapping, it's not possible to do a http request. What I wanted was to do a http request to get the referenced file from Sia Skynet and index data from that file.
This would result in less data on the blockchain, so a cheaper solution.
There is an IPFS package for The Graph, to get data from IPFS. A similar Sia package would be helpful.

GAS fees

A big limitation is that there has to be an on-chain action for The Graph to index. But on-chain transactions cost GAS and with high gas fees, there is no incentive to use this smart contract. Why would you pay gas to share a document?

As mentioned before, The Graph can already index data from IPFS. For example when an IPFS hash is posted to a smart contract, this hash will be available in your mapping. But this needs an on-chain transaction.

It would be really cool if The Graph could index immutable data on IPFS or Sia without an on-chain transaction. This would make The Graph useful for a lot more web3 scenario's.

This is not an easy problem the solve. Even though data on IPFS or Sia is immutable. It makes the process non-deterministic. The network may fail, or IPFS data is not available anymore.

A possible implementation from a developer perspective could be something like this:
The Graph could create an API endpoint that accepts IPFS or Sia hash/file locations.
It then fetches the data and processes the data like it was a smart contract event. Maybe store this data on a private (sub)blockchain, so it's similar to how The Graph indexes other blockchains. This can eliminate the need for on-chain transactions with gas fees.

Example

Send file hash to The Graph API
POST: https://thegraph.network/api/add
Data: sia://hash
Load data from sia://hash
Data is stored on a special The Graph blockchain, fees can be prepaid in GRT?
Mapping file gets the data on the blockchain as input and allows you to create subgraphs

More TheGraph metadata in Git

I like to store as much data as possible in source control / git. It would be nice if the description/logo and also example queries could be added to subgraph.yaml or another file that gets deployed to The Graph

How to send a secret to another Ethereum address?

All the data on the Ethereum blockchain and on a subgraph from The Graph is public. So how do we share a secret encryption key and a private key through these layers?
We want to share a document with another Ethereum address. Unfortunately, we don't know the public key that corresponds to another Ethereum address. If we did, we could use public/private key encryption.
One option would be that each Ethereum users first publishes their public-key. Then we can index that and then encrypt using their public key if we want to share data with that Ethereum address. It would create a big hurdle te be able to use this functionality.
I choose to introduce a central server that keeps the secrets. This server encrypts and stores the secrets and decrypts the secret only if you identified using your MetaMask signed string. It's secure, but introduces a central component to this solution.

Sia Skynet

During development, there were times that Skynet was slow. Especially at saving data. There was also a DDOS attack on multiple Sia Skynet endpoints. During that DDOS attack I was unable to develop. Decentralization did not fix these issues yet.

What's next

I think we are at a great point in time. It's starting to become possible to develop real distributed Apps, some infrastructure is ready, and a lot is being developed. But we are still early. Everybody is building, and new infrastructure is being developed every day.

I think it's really important to contribute to that and that's why all of the tools I'm developing, can be used as building blocks for others apps.
I'm developing SkyDocs with Microsoft .Net and C#. I'm using Blazor to compile my app to WebAssembly. This makes it possible to use static file hosting, like Sia Skynet.

The MetaMask login I've created as part of this grant for SkyDocs, is open source for other developers to use.
The Skynet API client I developed, is also open source and ready to use by other developers.
I've created a subgraph to share files for SkyDocs, but I've made it generic, so other apps can use the subgraph.
And there is a C# SDK to communicate with The Graph APIs.

Everything I'm doing can be used as a building block for other developers. It will save them time and they are able to create the dApps of the future.

SkyDocs will continue to be updated and function as a testing ground for other distributed platforms.