https://www.sitepoint.com/ipfs-swarm-decentralized-content-publication-storage/
In this article, we outline two of the most prominent solutions for decentralized content publication and storage. These two solutions are IPFS (InterPlanetary File System) and Ethereum’s Swarm.
With the advent of blockchain applications in recent years, the Internet has seen a boom of decentralization. The developer world has suddenly gotten the sense of the green pastures that lie beyond the existing paradigm, based on the server–client model, susceptible to censoring at the whims of different jurisdictions, cloud provider monopolies, etc.
Turkey’s ban of Wikipedia and The “Great Firewall of China” are just some examples. Dependence on internet backbones, hosting companies, cloud providers like Amazon, search providers like Google — things like these have betrayed the initial internet promise of democratization of knowledge and access to information.
As this article on TechCrunch said two years ago, the original idea of the internet was “to build a common neutral network which everyone can participate in equally for the betterment of humanity”. This idea is now reemerging as Web 3.0, a term that now means the decentralized web — an architecture that is censorship proof, and without a single point of failure.
As Gavin Wood, one of Ethereum’s founders, in his 2014 seminal work on Web 3.0 put it, there is “increasing need for a zero-trust interaction system”. He named the “post-Snowden web”, and described four components to it: “static content publication, dynamic messages, trustless transactions and an integrated user-interface”.
Decentralized Storage and Publication
Before the advent of cryptocurrency — and the Ethereum platform in particular — we had other projects that aimed to develop distributed applications.
- Freenet: a peer to peer (p2p) platform created to be censorship resistant — with its distributed data store — was first published in 2000.
- Gnutella network: enabled peer-to-peer file sharing with its many client incarnations.
- BitTorrent: was developed and published as early as 2001, and Wikipedia reports that, in 2004, it was “responsible for 25% of all Internet traffic”. The project is still here, and is technically impressive, with new projects copying its aspects — hash-based content addressing, DHT distributed databases, Kademlia lookups …
- Tribler: as a BitTorrent client, it added some other features for users, such as onion routed p2p communication.
Both of our aforementioned projects build on the shoulders of these giants.
IPFS
The InterPlanetary File System was developed by Juan Benet, and was first published in 2014. It aims to be a protocol, and a distributed file system, replacing HTTP. It’s a mixture of technologies, and it’s pretty low level — meaning that it leaves a lot to projects or layers built on top of it.
An introduction to the project by Juan Benet from 2015 can be found in this YouTube video.
IPFS aims to offer the infrastructure for reinventing the Internet, which is a huge goal. It uses content addressing — naming and lookup of content by its cryptographic hash, like Git, and like BitTorrent, which we mentioned. This technique enables us to ensure authenticity of content regardless of where it sits, and the implications of this are huge. We can, for example, have the same website hosted in ten, or hundreds of computers around the world — and load it knowing for sure that it’s the original, authentic content just by its hash-based address.
This means that important websites — or websites that may get censored by governments or other actors — don’t depend on any single point, like servers, databases, or even domain registrars. This, further, means that they can’t be easily extinguished.
The Web becomes resistant.
One more consequence of this is that we don’t, as end users, have to depend on internet backbones and perfect connectivity to a remote data center on another continent hosting our website. Countries can get completely cut off, but we can still load the same, authentic content from some machine nearby, still certain of its authenticity. It can be content cached on a PC in our very neighborhood.
With IPFS, it would become difficult, if not impossible, for Turkey to censor Wikipedia, because Wikipedia wouldn’t be relying on certain IP addresses. Authentic Wikipedia could be hosted on hundreds or thousands of local websites within Turkey itself, and this network of websites could be completely dynamic.
IPFS has no single point of failure, and nodes don’t need to trust each other.
Addressing the content is algorithmic — and it becomes uncensorable. It also improves the efficiency. We don’t need to request a website, or video, or music file from a remote server if it’s cached somewhere close to us.
Learn PHP for free!
Make the leap into server-side programming with a comprehensive cover of PHP & MySQL.
Normally RRP $39.99 Yours absolutely free
This can eliminate request latency. And anyone who’s ever optimized website speed knows that network latency is a factor.
By using the aforementioned Kademlia algorithm, the network becomes robust, and we don’t rely on domain registrars/nameservers to find content. Lookup is built into the network itself. It can’t be taken down. Some of the major attacks by hackers in recent years were attacks on nameservers. An example is this particular attack in 2016, which took down Spotify, Reddit, NYT and Wired, and many others.
IPFS is being developed by Protocol Labs as an open-source project. On top of it, the company is building an incentivization layer — Filecoin — which has had an initial coin offering in Summer 2017, and has collected around $260 million (if we count pre-ICO VC investment) — perhaps the biggest amount collected by an ICO so far. Filecoin itself is not at production-stage yet, but IPFS is being used by production apps like OpenBazaar. There’s also IPFS integration in the Brave browser, and more is coming …
The production video-sharing platform d.tube is using IPFS for storage, while Steemit is using it for monetization, voting, etc.
It’s a web app that’s waiting for wider adoption, but it’s currently in production stage, and works without ads.
Although IPFS is considered an alpha-stage project, just like Swarm, IPFS is serving real-world projects.
Other notable projects using IPFS are Bloom and Decentraland — an AR game being built on top of the Ethereum blockchain and IPFS. Peerpad is an open-source app built to be used as an example for developers developing on IPFS.
Swarm
According to Viktor Tron, of the Ethereum Foundation, “basically, Swarm is BitTorrent on steroids”.
Swarm, by Ethersphere, aims to solve the same problems as IPFS. According to its GitHub page —
Swarm is a distributed storage platform and content distribution service, a native base layer service of the Ethereum Web 3 stack. The primary objective of Swarm is to provide a sufficiently decentralized and redundant store of Ethereum’s public record, in particular to store and distribute Đapp code and data as well as block chain data.
Viktor Tron is currently behind Swarm as its lead developer. He was one of the first employees of the Ethereum Foundation. Ethereum Foundation is funding the project development, along the lines of Gavin Wood’s vision of Web 3.0 that we quoted. So, Swarm is more integrated with the Ethereum ecosystem, and along with Whisper and Ethereum Virtual Machine, it’s aiming to build a next-generation platform for distributed apps, or Đapps.
Swarm is in an earlier stage of development than IPFS. To quote Viktor Tron —
IPFS is much further along in code maturity, scaling, adoption, community engagement and interaction with a dedicated developer community.
Once Swarm becomes production-ready, it will provide an incentivization layer and integration with Ethereum’s smart contracts, which should give plenty of room for creativity and innovative applications.
Neither the incentivization layer of Swarm nor of IPFS (Filecoin) are currently ready for use.
Note: at the time of writing (May 2018), Swarm’s lead developer has announced the release of POC3, which keeps its roadmap on the clock, and gives reasons for optimism regarding Swarm becoming production-ready in 2018.
While IPFS aims to build a protocol, and is a lower-level, more generic project, Swarm ties into the Ethereum’s Web 3 vision, with more focus on censorship resistance: it “implements plausible deniability with implausible accountability through a combination of obfuscation and double masking”.
This reminds us of the Freenet project, where those hosting certain content don’t necessarily have access to it, or know what it is.
Swarm, with its incentivization mechanisms, is aiming to provide higher level solutions. It —
exploits the full capabilities of smart contracts to handle registered nodes with deposit to stake. This allows for punitive measures as deterrents. Swarm provides a scheme to track responsibilities making storers individually accountable for particular content.
Compared to IPFS, Swarm has a lot of focus on these mechanisms. On the one hand, this includes incentives for long-term storage of not-so-popular content, and on the other, incentives for highly popular, high-bandwidth content. These two require two different approaches to penalties/rewards.
In Swarm’s case, this requires working on cryptographic constructs known as Proof-of-Custody, which make it possible “to have a compact proof, proving to any third party that you store a blob of data without transferring the whole data and without revealing the actual contents”. So proving a storage of some content doesn’t require the full download of that content every time.
Swarm even has an Accounting Protocol, SWAP, currently in development, as one level of incentivization.
Currently, before incentivization mechanisms are published, which is expected to happen in 2018, Swarm functions like a cache: less popular content can get deleted, and there’s no insurance against that.
Swarm will be usable as cloud hosting, while IPFS relegates this to projects that will be built on its infrastructure. IPFS leaves it to the implementors/developers to find the actual storage devices.
IPFS itself, as lower layer, has no guarantees of storage. While Swarm includes this in its roadmap, the IPFS team, in comparison, plans this on the Filecoin level, but it’s just in idea stage at the moment.
There’s a two-part YouTube interview with Tron where he explains the Swarm project in less technical terms:
There are two projects that build further on IPFS and Swarm that are worth mentioning in the context of Đapps: distributed applications. Since both projects allow for only a limited level of dynamic content, database-oriented projects built on top of these distributed systems add significant value.
OrbitDB is a “serverless, distributed, peer-to-peer database” that uses IPFS for its data storage.
It’s a database that works both for Node.js and in browsers. Its development is active, and is being sponsored by Protocol Labs. After its $260 million fundraising in 2017, the future of OrbitDB — just like that of IPFS — looks promising.
OrbitDB is part of the Node.js/npm ecosystem.
Wolk is a project/token that’s building a database — SWARMDB — using Swarm’s codebase. Behind it is a Californian startup, Wolk Inc., that managed to raise around 7,100 ETH in its ICO in 2017. WOLK promises a censorship-resistant distributed database powered with WLK token as its incentivizing layer. It provides a Go, Node.js and HTTP interface.
They claim Swarm and Bancor as their partners.
While it’s hard to predict success and adoption of these projects, or ascertain their quality, as IPFS and Swarm progress and become more production-ready and reach wider adoption, it’s pretty certain we’ll see more projects like these.
Swarm’s Orange Paper is an interesting, albeit a very technical read.
A longer comparison of the two projects can be found here.
Commonalities
Things that both IPFS and Swarm share are hash-based content addressing, which we described before. And while this provides git-level version control of the content, hosted on both systems, and censorship-resistance, deleting the content is something that remains to be solved.
Immutability provides guarantees of authentic content, but changes to the content produce new addresses, so to provide editing capability, additional layers are necessary.
From the perspective of different web apps, both projects support only static content. So, there’s no back-end apps with interpreted languages, like PHP, Python, Ruby, or Node.js. For Swarm, this is where EVM comes into play, but EVM also has its own inherent limitations.
Conclusion
Both IPFS and Swarm are promising projects, although one can’t help but wonder if the developers have set overly ambitious goals. If they succeed with their development roadmaps, and achieve wider adoption, there’s no doubt this will bring big changes to the Internet as we know it.
Tonino is a web developer and IT consultant who's dived through open-source code for over a decade. He's also a crypto enthusiast, Linux fan, and moderate libertarian.
New books out now!
Get practical advice to start your career in programming!
Master complex transitions, transformations and animations in CSS!
Latest Remote Jobs
Junior Application Developer
DonorDrive
- sql
- javascript
Senior Android Developer
Alertive
- kotlin
- java
Senior Data Engineer
Goodway Group
- node
- python
Solution Engineer
Data Virtuality GmbH
- sql
- linux
Drupal/PHP Engineer
Nebo
- drupal
- php
Things to remember in a stabilising Crypto market
By Simon Julian
September 5, 2018Share:
Free JavaScript Book!
Write powerful, clean and maintainable JavaScript.RRP $11.95
At the end of January 2018 cryptocurrencies big and small did what they’ve been doing for the past few years now and took a (pretty massive!) dive. The market went from an all-time high of around $USD830 billion to around $USD240 billion at the time of writing.
Fair to say that a lot of people lost it, a lot of people got burnt and a huge number of (fairly undeserving) blockchain projects lost anything up to 90% of their value. Some deserving ones lost all their value – which was a good thing in the end. Ultimately the crash, particularly for Bitcoin, was another round in a pattern that has been in place for the past few years. Not that anyone who invested in Bitcoin or many of the legitimate Altcoin projects in December of 2017 would have been comforted by that.
As Bitcoin and the market in general begins to show signs of a recovery it’s worth looking at a few reasons why a market downturn isn’t always the end of the world.
Adoption!
One thing to always keep in mind with Blockchain (and anything to do with cryptocurrencies) – it is VERY early days. The market to this point has been made up of what is still a relatively small number of retail investors, for all of the hype that it has drawn.
Over the past 12 months we’ve seen a huge upturn in the adoption of blockchain in many areas of business – we’re beyond individuals and corporations and now into entire countries looking at blockchain adoption and usage. As many of the current major players (South Korea, China and others) move beyond the first stages of regulation and opening new markets for projects and therefore cryptocurrencies to grow, adoption will begin to increase exponentially.
The higher the level of acceptance and therefore legitimacy for real uses in Blockchain the more valuable the assets and therefore the whole market become. The addition of Bitcoin ETF’s and other asset trading vehicles (when they arrive, and they’re not here yet!) will further push adoption, possibly not all of it positively.
Market downturns
Crash is a nasty word, so let’s call it a downturn. In crypto they tend to happen regularly, firstly with Bitcoin and now with Ethereum and the Altcoin markets. Cryptocurrency markets are incredibly volatile, in the main because up to now they have been retail and small investor-driven – meaning it is very easy to create a panic and cause major issue – FUD is a prevalent concern in cryptocurrency markets and even very simple whispers of issues in a project can cause major price changes . Of course, with a dearth of fraudulent ICO’s and a LOT of very under-resourced, poorly managed projects out there, the rumour can also be entirely true!
The important things to remember about market downturns are (a) they’re not new, and they happen in every market and (b) what you do when the market turns south can determine whether it is a positive or a negative for you. Leading to the next point…
Your money isn’t lost unless you sell
Unfortunately the fear of missing out in any sort of trading is real, and human nature is in general pretty unhelpful in the kinds of scenarios you find yourself in when the market turns. For the majority of new traders, especially in markets like cryptocurrency where so much media attention talks to massive riches and incredible returns, watching the value of a portfolio shrink, sometimes by large percentages in short time periods, can be a panic-stricken experience.
Two rules should always apply.
The first is don’t put in more than you can afford to lose. It’s a simple concept but one that people continually forget – if you can afford to lose it and the market turns down you’re not going hungry or losing your home, which makes everything else easier to handle.
The second is ‘don’t flinch’ – rational decisions are better than panicked ones. Made some profit and worried that the market is turning? Sell and get out ahead. Market turned on you and you’re in loss-making territory? Followed rule #1? Sit and wait – in a lot of cases if the project is good the market will turn again and you’ll have another chance to profit.
Down means opportunity
Google “Buy the dip” – there are lots of great pieces of advice out there and good trading strategies that go way beyond an adage. Dips are opportunity, plain and simple. If you look at them that way, and you’ve done the research on the projects that you are buying into, then a market turn can be a great way to make bank.
There is a lot more to it than what it sounds though, and buying the dip in a micro sense is a great way to end up with no money – for early traders or people who are just taking a punt, do the research, buy when the market is well down and then sit on your asset – there are opportunities in patience too!
Play safe!
If you’d like to join the conversation or add some thoughts you can do so below.