Decentralised File Storage using Web3.Storage, IPFS and Filecoin in a Jiffy!

Decentralised File Storage using Web3.Storage, IPFS and Filecoin in a Jiffy!

·

10 min read

IPFS and Decentralised Network

So as Richard Hendricks portrayed the decentralisation concept on a famous television show Silicon Valley, are we still relying on centralised tech companies to store and manage our information? Or are you going to prefer decentralisation to absorb the importance of its features and change the power dynamics between tech companies and their users?

credits: Pied Piper Page,

So let’s assume, you want to store all your cat videos and images on a decentralised web. Hell yeah, what If I tell you that you can not only limit the storage of these files to one location but can explore the possibility to store this data that is spread out across a vast network of computer systems connected via the internet?

Also, Data in IPFS is stored in 256kb IPFS objects. These objects then create a link between them and connect to all the chunks of the respective data, but don’t worry your cat is fine.

So technically what we are doing to your “cat” is that we are running the cryptographic hash functions to produce a digest from the raw data and that in return gets us a Content Address/Identifier which helps the IPFS to locate this file or say your cat on the network.

credits: Carson Farmer medium blog,

What’s the hype and pros of using Web3.Storage

Web3.storage is the latest addition to the decentralized technology landscape, offering developers a seamless and frictionless experience when it comes to storing and accessing data on a decentralized network. This innovative platform harnesses the power of decentralized storage solutions, such as Filecoin and IPFS, to provide a secure and reliable solution for developers building next-generation applications and websites.

By leveraging web3.storage, developers can reap all the benefits of decentralized storage without having to navigate the technical complexities typically associated with these solutions. All that is required to start using web3.storage is an API token and your data, making it a convenient and straightforward option for developers.

Under the hood, web3.storage is powered by the provable storage of Filecoin, ensuring that data is stored securely on the decentralized network. Additionally, data is made accessible to users over the public IPFS network, providing a robust and scalable solution for developers building decentralized applications.

credits: web3.storage

In conclusion, web3.storage simplifies the process of building on decentralized technologies by providing a user-friendly interface and eliminating the need for developers to have a deep understanding of decentralized storage solutions. This makes web3.storage an attractive option for developers looking to build next-generation applications and services in a secure and reliable environment.

IPFS (InterPlanetary File System) is a decentralized file storage system that allows for the storage and retrieval of data in a distributed network. IPFS aims to replace traditional centralized servers with a network of peer-to-peer nodes that can store and serve data more efficiently and securely. IPFS is an identity protocol for storing and sharing data on the distributed web. It’s free and open-source.

Filecoin is a decentralized storage network that uses economic incentives to ensure reliable file storage. It operates on a peer-to-peer network, allowing anyone to participate as a storage provider or file owner. The pricing and availability of storage are determined by market forces, creating a fair and transparent system for file storage. Filecoin offers a secure and reliable alternative to traditional centralized storage solutions.

credits: Ethan Wu medium blog

For more understanding of filecoin and IPFS visit the blog, Understanding IPFS / FileCoin by Ethan WU

Use case and what’s there for you

Web3.storage and IPFS can be used in combination with Livepeer to store and playback video in a decentralized manner. Livepeer is a decentralized video infrastructure platform that allows for live streaming and video transcoding. Here are some potential use cases for this setup:

  1. Decentralized video content delivery: Video content can be stored on web3.storage and IPFS and served through the Livepeer network, allowing for decentralized and highly available video content delivery.

  2. Tamper-proof video storage: Video content stored on web3.storage and IPFS is secured by cryptographic hashes, making it tamper-proof and ensuring the integrity of the stored video files.

  3. Live streaming and video on demand: Livepeer can be used to facilitate live streaming and video on demand, with the video files stored on web3.storage and IPFS. This allows for highly scalable and secure video delivery.

Overall, the combination of web3.storage, IPFS, and Livepeer can provide a powerful solution for decentralized video storage and playback, offering improved scalability.

credits: messari

How we Implemented it

Introducing web3.storage implementation, the solution to all your file storage needs for your projects, websites or applications. With web3.storage, there's no need to worry about complex details or struggles. All you need to get started is a free API token.

The foundation of web3.storage lies in the Filecoin network, a proven and reliable file storage solution, and its data is made accessible through the public IPFS network. But as a user, you don't have to understand the intricate workings of these technologies. web3.storage simplifies the process, making it straightforward to store your data on the decentralized web.

Whether you're a seasoned developer or just starting, web3.storage offers a simple and frictionless experience, while still delivering the benefits of decentralized storage. Say goodbye to the hassles of centralized storage solutions and enjoy the peace of mind that comes with knowing your data is secure and tamper-proof on the decentralized web.

I have explained the work and implementation with all possible code snippets, just follow along and at the end of the blog, you’ll have all your files over the decentralised network.

Prerequisites:

Required Node version 14 or higher and NPM version 7 or higher. Check your local versions.

node --version && npm --version
> v16.4.2
> 7.18.1

1. Setup:

Create a new account using GitHub or email to get the API Token and manage your data. This API Token lets you interact with web3.storage services.

Set up a new node project using npm init to work along the code snippets and explanations provided in the blog.

  • Generate an API Key from web3.storage website, we’ll be needing this for storing the files
API_KEY = <YOUR API KEY>
  • Package.json for the node file, copy and paste this into your project and install these dependencies using npm or yarn.
{ 
    "name": "web3-storage-quickstart", 
    "version": "0.0.0", 
    "private": true, 
    "description": "Get started using web3.storage in Node.js", 
    "type": "module", 
    "scripts": { 
    "test": "echo \"Error: no test specified\" && exit 1" 
   }, 
    "dependencies": { 
    "minimist": "^1.2.5", "web3.storage": "^3.1.0" 
   }, 
    "author": "YOUR NAME", 
    "license": "(Apache-2.0 AND MIT)" 
}
  • Download the dependencies above and web3.storage
npm install 
npm install web3.storage

1.1 Upload Image and Metadata JSON:

  • We need to get the image in the same directory, that is to be uploaded.

  • Here we upload the image first and get its CDN.

  • Then using the image CDN we create an Image URL using a gateway URL function we have made.

  • Embed this URL in the metadata.json file that we create from objects to JSON using stringify

  • Then we upload this JSON file to the IPFS and in return get its CDN and URL

  • Now create uploadFunc.js, or whatever you want to name your js file. the code is well commented on and explains each line.

import { getFilesFromPath, Web3Storage } from "web3.storage";
import fs from "fs";
// import dotenv from 'dotenv'
// dotenv.config()

/* ********************** Token ********************** */
const apiToken = <YOUR API TOKEN>

/* ********************** Image ********************** */
/* Function gateway URL for image */
function makeGatewayURLImage(imgCID, imgName) {
  return `https://${imgCID}.ipfs.w3s.link/${imgName}`;
}

/* Construct client with token and endpoint */
const client = new Web3Storage({ token: apiToken });

/* Get file from path for image */
const img = await getFilesFromPath("./img.jpg");
/* Get image name from path */
const imgName = img[0].name;

/* get imgCID from web3.storage */
const imgCID = await client.put(img, { name: "imgName" });
console.log("imgCID: ", imgCID);

/* Make gateway URL for image */
const imgURL = makeGatewayURLImage(imgCID, imgName);
console.log("imgURL: ", imgURL);

/* ******************** Metadata JSON ******************** */

/* Function for gateway URL for json */
function makeGatewayURLJSON(jsonCID, jsonName) {
  return `https://${jsonCID}.ipfs.w3s.link/${jsonName}`;
}

/* metadata for json */
const metadata = {
  name: "TestJSON",
  description: "Uploading json with imgURL embedded",
  image: imgURL,
  attributes: [
    { trait_type: "Participant", value: "Amy" },
    { trait_type: "Event Name", value: "TestEvent" },
    { trait_type: "Date", value: "27-12-2022" },
  ],
};

/* conversion to json file using fs*/
fs.writeFileSync("metadata.json", JSON.stringify(metadata));

/* Get file from path for json */
const json = await getFilesFromPath("./metadata.json");
/* Get json name from path */
const jsonName = json[0].name;

/* get jsonCID from web3.storage */
const jsonCID = await client.put(json);
console.log("jsonCID: ", jsonCID);
/* Make gateway URL for json */
const jsonURL = makeGatewayURLJSON(jsonCID, jsonName);
console.log('jsonURL: ',jsonURL);
  • Run the upload javascript file code by,
node uploadFunc.js

We get CDNs and URLs in the terminal, these are the respective file’s CDN and URLs generated for JSON Metadata and Image. The URLs can be used to access the files stored on the IPFS Network, HURRAH AND KUDOS, you did it!

1.2 Upload a video to IPFS Network:

  • We use the same method and get the video to be uploaded to the same directory as the root file.

  • Then it's the same process as we upload the image

  • uploadFunction.js is the function for uploading the video, the code is well-commented and explains each line.

import { getFilesFromPath, Web3Storage } from "web3.storage";

/* ********************** Token *************************** */

const apiToken = <YOUR API TOKEN>

/* *********************** Image ************************** */

/* Function gateway URL for image */
function makeGatewayURLImage(vidCID, vidName) {
    return `https://${vidCID}.ipfs.w3s.link/${vidName}`;
  }

/* Construct with token and endpoint */
const client = new Web3Storage({ token: apiToken });

/* Get file from path for image */
const vid = await getFilesFromPath("./CatVideo.mp4");
/* Get image name from path */
const vidName = vid[0].name;

/* get imgCID from web3.storage */
const vidCID = await client.put(vid, { name: "vidName" });
console.log("vidCID: ", vidCID);
/* Make gateway URL for image */
const vidURL = makeGatewayURLImage(vidCID, vidName);
console.log("vidURL: ", vidURL);
  • Run the upload javascript file code by,
node uploadFunction.js

Again, we get the respective CDN and URL of the video uploaded to the IPFS Network. You did it again mate well done.

1.3 Uploading multiple files:

Here the code doesn't change much, only the part where we upload the file, we pass it as an array buffer in [ file1, file2, …. fileN] Square brackets.

If we want to upload metadata, video or say image simultaneously we can do that too.

  • we’ll upload them to a single CID.

  • but have the specific URLs to the files as child URLs.

  • Also, we’ll have the parent URLs of those files.

import { getFilesFromPath, Web3Storage } from "web3.storage";
import fs from "fs";

/* **************** Token and Client ************************* */
const apiToken = <YOUR API TOKEN>

/* Construct client with token and endpoint */
const client = new Web3Storage({ token: apiToken });

/* ***************** Video + JSON Files******************** */

/* metadata for json */
const metadata = {
  name: "Video Title",
  description: "Video Description goes here",
};

/* conversion to json file using fs*/
fs.writeFileSync("metadata.json", JSON.stringify(metadata));

/* Here we are uploading multiple files through an array specified */
const main = await getFilesFromPath(["./metadata.json", "./video.mp4"]);

/* Name of file */
const jsonName = main[0].name;
const videoName = main[1].name;

/* **************** URLS Functions to be called **************** */
function parentURL(CID) {
  return `https://${CID}.ipfs.w3s.link`;
}
function videoURL(CID, videoName) {
  return `https://${CID}.ipfs.w3s.link/${videoName}`;
}
function jsonURL(CID, jsonName) {
  return `https://${CID}.ipfs.w3s.link/${jsonName}`;
}

/* ****************** Upload to Web3 Storage ******************* */

/* get CID from web3.storage */
const CID = await client.put(main);
console.log("CID: ", CID);

/* ****************** Generate and console URLS ***************** */

/* get Parent URL from CID */
const parent = parentURL(CID);
console.log("Parent URL: ", parent);

/* get Video URL from CID */
const video = videoURL(CID, videoName);
console.log("Video URL: ", video);

/* get JSON URL from CID */
const json = jsonURL(CID, jsonName);
console.log("JSON URL: ", json);

Kudos guys, you did it, you can again run this file in node. You’ll get the parent URL containing all the files and the specific child URL containing each Metadata and Video respectively.

Conclusion

In conclusion, web3.storage and IPFS offer a powerful and accessible solution for decentralised file storage. With web3.storage, users can enjoy the benefits of this technology without having to worry about the underlying complexities. Whether you're a seasoned developer or just starting, web3.storage makes it easy to store your files securely on the decentralised web.

The Documentation of Web3.storage is also a great help and I’d recommend following it since I followed it too. Found it to be on point and super precise. The link is attached below.
Web3 Storage Documentation

Did you find this article valuable?

Support Mohd Sameer by becoming a sponsor. Any amount is appreciated!