DEV Community: Functionland

FxExchange: Functionland's DAG Exchange Protocol

Masih H. Derkani — Sun, 19 Feb 2023 12:02:05 +0000

Every piece of information in the Functionand Data Protocol is encrypted and shielded before it leaves your device. That's how we do things and we wouldn’t have it any other way. But how does it all work? What's really going on when you take a photo and store it on FxFotos app, or save your notes in FxFiles?

The scene

In a typical Web 2 setup, information produced by the user is uploaded to a centralized server. This information is typically encrypted on transit, but often stored on the server side unencrypted. In cases where it is encrypted at rest, the server administrators, operators or app developers often have access to the unencrypted user's data.

That doesn’t sound good, does it? But wait: it gets worse. Aside from accessibility, the integrity of the data itself is typically not verified. For example, there is nothing that can stop an administrator from changing a pixel from red to maroon in a picture that the user has taken and uploaded onto a centralized server. Web 2 has no inherent mechanism safeguarding users from data tampering or alteration.

In a decentralized web, information is stored differently. The data is divided up into interlinked smaller pieces that typically make up a Directed Acyclic Graph (DAG) with pieces scattered across multiple peers. The use of the word "peer" is intentional. Unlike servers, peers are “corporate” in the truest sense of the word: they are united to work together as a single entity. Typically peers share equal responsibilities and importance. Each piece of individual data is verifiable: it is identified by its cryptographic hash and linked to other pieces that are also identified by their cryptographic hash. The thing about cryptographic hashes is that while they are very, very sensitive to changes in data, if given the same data they will always return the same hash. This means if the data is addressed by its hash, then we should get the exact same data. Having asked for data by its hash, the receiver can calculate the hash after having received the data and compare it with the original hash they were looking for.

This property is at the heart of a decentralized web; it is a simple yet powerful property that allows much more powerful concepts to be built up on top. Data in the decentralized web is not always encrypted, but it could easily BE encrypted. Today this aspect combined with built-in verifiability features has fostered a fundamental rethinking of data ownership, sovereignty and privacy.

At this point it’s fair to ask: But does all of this dividing, hashing and interlinking make things more complicated to work with? After all, an app that doesn’t work is no app at all! Photos need to get stored, files need to get archived, and notes need to get saved.

FxExchange Protocol

Enter FxExchange Protocol: Functionland's bread and butter when it comes to DAG exchange. FxExchange is part of the Functionland Data Protocol: given a DAG, the protocol’s job is to quickly and efficiently shift it from one point to another. At its core, the Functionland Data Protocol uses Web Native File System (WNFS) to store your files.

WNFS offers a familiar lens through which to deal with data: UNIX file system files and directories, while at the same time offering a way to encrypt and obfuscate it all into a nifty thing called the “Dark Forest.” (And yes, that’s as cool as it sounds!) As the user interacts with a virtual drive of files and directories, the entire data is encrypted and transformed into a DAG, the shape of which has no resemblance to the original directory structure. This DAG is then sent into the Functionland network for sharing and backup. No-one, except for the original user, can decipher the DAG or see metadata about the original, user-created files or directories.

It also goes without saying that data integrity is preserved by the good old fashioned hashing not only in its encrypted form, but also in its unencrypted form at the originating device.

As soon as one such DAG is generated, the app signals to Functionland’s Data Protocol to sync it to Blox for backup. This is where FxExchange takes over.

FxExchange is built on top of the vanilla GraphSync protocol: a simple protocol for exchanging DAGs over libp2p. It is heavily used by FileCoin Network through another protocol called Data Transfer for exchanging data. In addition, it natively understands DAGs represented in IPLD. This allows the use of sweet IPLD features - Selectors are just one example…but we’ll be taking a look at those in another blog post. (Stay tuned for more!) To learn more about GraphSync, take a look at this document.

FxExchange introduces specific features to control the DAG exchange; namely:

authorization of exchange between two or more libp2p peers
authorization management APIs
push or pull information to/from libp2p peers
a thin mobile client implementation compatible with gomobile for use in mobile apps
sync only the absent fraction of a DAG, thanks to IPLD selectors, and finally
manage multiple WNFS drives that may belong to different users.

Using libp2p as the underlying transport allows FxExchange to continue to function, even in the face of poor connectivity. It also allows FxExchange to opportunistically select the best path for exchanging bytes depending on a given environment. Thanks to built-in support for NAT discovery and Hole Punching built natively into libp2p, FxExchange is able to function on pretty much any network independent of centralized servers. In cases where direct connection is not possible, FxExchange uses the latest libp2p relay feature to transitively connect through other peers in the network.

The control APIs in FxExchange use HTTP-compatible wire protocol for extensibility, sent and received over libp2p go-stream. This offers a familiar interface to developers who wish to extend the FxExchange features, given that most developers are deeply familiar with the HTTP protocol. Concurrently, it enables developers to swap libp2p for vanilla HTTP should the need arise. (This can be useful in cases where FxExchange has to sit behind pre-existing HTTP routing systems.)

FxExchange protocol works hand-in-hand with the FxBlockchain protocol to deliver Blockchain-Attached Storage (BAS) with privacy built right into its core. Stay tuned for future blog posts as we dive deeper into the exchange and blockchain interaction!

Ditching .NET and finding faith again after 10 years!

Keyvan M. Sadeghi — Tue, 13 Dec 2022 14:05:43 +0000

TL;DR .NET MAUI is a dope multiplatform stack and may well be the perfect candidate for your next app. Don’t dismiss it because it’s “Made in Microsoft.” You can check out Fx Files as a testimonial to its interop with JS, and the general new .NET direction.

My way or the highway! That's the prevalent mantra in Big Tech. Even from the grave, Steve Jobs’s guideline of “Be Incompatible!” is haunting us to this day — when we have a guest over and they can’t charge their phone with our cable! Worse yet, there’s always a “cult” of followers, belittling others who don’t fall in line under the same banner, constantly pointing out tech flaws from the other side, and they are so susceptible to confirmation bias from their peers. The same toxic pattern can sometimes be found in developer circles too.

Microsoft was no different under Steve Balmer (nor Gates). I should know because I was a die-hard Microsheep up until 2012 🙈. You had the Microsoft Universe to drown in, and to be fair it was quite compelling! Don’t you pity those poor backwards fellas who have to open a terminal to do the simplest things? Need an IDE? Behold the mighty Visual Studio! Need UI? Here you have WPF, putting the frontier of UX dev at your fingertips! And it’s all drag and drop, with seamless codegen that you can witness live! Also bear in mind, REAL software runs exclusively on Windows, do you know of a single enterprise app that’s elsewhere?

Time would prove all wrong, but back then these arguments seemed sound.

Microsoft is a fear factory. Oh, your team is still not Agile? You have to use our software suite that gives MANAGEMENT measurable VISIBILITY on what happens in your org!

During my studies, classmates were doing their homework in spaghetti C, while I swaggered by and showed off my shiny C# class diagram and glorious WinForms UI! I had a date with destiny though. The first job I landed was at a research lab in Hong Kong. Had a 2-week deadline to deliver a library written in Python, running on Linux, neither of which I’d ever touched! No longer did I have the luxury of being like “Pff, this weird shitty open-source stuff again!” The date was set and I needed to push my code into this other weird thing: GitHub. Thus began my journey. In the decade since, I delved deep into all that open-source stuff, convinced every employer to publish pieces of my work as open-source, ditched languages and frameworks left and right, and learned to be loyal to nothing if not innovation!

Given this history, I was highly skeptical when my friend mehrandvd pitched me .NET for development of our next app. We had, one might say, unrealistic expectations: a File Manager that runs on Android, iOS, Windows and macOS. It should handle local file operations like copy, rename, etc., as well as utilities like thumbnails, zip extraction and in-app file viewers. The same UI, we decided, should have another tab where you browse your “remote files.” These could be on any cloud provider, AWS S3 and such, or weirder places like IPFS. I.e. being “Backend Agnostic.” A developer implements an interface, submits the pull request, and the provider shows up as another configurable “source” for your files. We also wanted interoperability between one’s local and remote files, things like syncing a local folder on a remote provider.

Craziest of all, we wanted all these developed in just 3 months!

And I kid you not, Mehran and the team delivered! They even managed to squeeze in extra features like code highlighting! Here’s a peek:

Project on GitHub:
https://github.com/functionland/fx-files

And here’s the app on Google Play and Microsoft Store:

(iOS, Mac and remote file integration in an upcoming update.)

MAUI’s appeal is abstracting away capabilities of different platforms under the same API. For our use case, as an example, we needed to talk with four different file systems. MAUI offers these as a unified Storage API. Also in edge cases that require you to write platform-specific code, e.g. performance optimization for infinity scroll, you have the freedom to do so.

Another aspect that surprised me was the way Microsoft has started to play fair with others. MAUI code can work in tandem with other languages and frameworks in your stack. This was a hard requirement for us, since our IPFS integration is a polyglot of Go, Rust, JavaScript and other upstream 🐉s I’m not aware of. This is so unlike the Microsoft I knew! After all, they were once famous for their EEE strategy!

A few more data points like MAUI and I may revert to Microsoft believer again! Pivoting the corporate culture at this scale is nothing short of astonishing. I salute Satya Nadella and everyone involved in this shift. Eager to see more!

Your feedback is much appreciated as always. Hope to get acquainted with you in the comments section! ❤️

Google Photos open-source alternative with React Native

Keyvan M. Sadeghi — Thu, 03 Jun 2021 15:57:44 +0000

Greetings fellow DEVs! You've probably heard the news about Google Photos becoming a paid service starting this month. We've been eying this date for a while and were preparing something that we think is kinda cool! Drumroll please... Photos UI partially implemented in React Native! Here's a peek:

This implementation is backend-agnostic, you can fork the repo and implement an interface to connect it to your own backend! We're also working on an open-source p2p backend implementation, more on this later in the article.

If you like what you see, there are ways you can help us take it to the next level:

Star the repo on GitHub!
Join the discussions, tell us what you think about where it needs to go from here.
Contribute code! Photos is currently being maintained only by Ehsan, he's eager to welcome new collaborators!
We have lots of cool things planned, including a Kickstarter campaign! Join the mailing list to hear about all updates!
We're actively seeking funding, if you can help with this in any way, please drop us a message!
You can also follow us on Twitter, Telegram and YouTube.

This is still at prototype stage and not ready for prime time, but we wanted to share early to hear your valuable feedback and hopefully have those of you who are interested join us to build a community around an ambitious yet noble goal:

Ending Big Tech's reign by building open-source p2p apps!

Making Photos paid? Really? Haven't Google already made billions by training AI models from our pictures and videos? Us being the product was not enough, we should also pay now?

I don't want to sound arrogant or ungrateful. Truth be told, I personally owe Big Tech a great deal! The very ability to write these words for example, I wouldn't know them if it wasn't for Windows (and me being a nerd)! And I was able to use Windows because a version of it existed that probably even Microsoft wasn't aware of: MRT Windows! (MRT being the initials of the dude who created it)

Being born into a country disconnected from the global economy, we didn't have legitimate access to software such as Windows, but Mr. MRT had BitTorrent and lots of blank CDs to cater us :) p2p was the missing piece to the puzzle: although we had in effect unwarranted access to Microsoft's intellectual property, a whole generation of developers was raised thanks to (unlawful) democratized access to software. Instead of the stereotypic Jihadists that the world expected, these people turned out quite civilized, some of them who moved abroad are now working for the Big Tech.

A large population of US citizens, meanwhile, didn't (by enforced law) have free access to Windows. Those who ended up attacking the Capitol in January might've acted different had they been growing up with Windows as kids!

Hopefully the above example conveys the importance of free access to software, free both as in speech and as in beer!

Does a workable solution for delivering and maintaining free software actually exist?

Yes and no. There are architectural and operational issues. We can, however, arrive at a solution by refactoring to Web 3.0! It helps us address several challenges:

1. Servers are expensive, how to pay for them without charging the users?

True, any application with enough users comes with a big monthly AWS bill. But why isn't this an issue for dev tools? We've had top notch UIs even for niche use cases such as state management in a specific library. Duh! These don't need a server :) our dev machines are the server for these tools. Hmm... developers have their own servers, right...

What if users had their own "Personal Server" too? This model has worked for Plex as an example, how about taking it one step further? Every house has a fridge for keeping food, would it make sense to also have a "store of data"? This is the main idea here, there's a niche who already have PCs at home (PC gamers, people still keeping their old PCs, etc.), we can write software that turns a PC to a server for a household's data needs.

In reality, the leverage that the Big Tech holds us hostage over is:

For sure, serving the world population takes acres upon acres of data centers. But for a household? Wouldn't a Raspberry Pi and a few TB of HDD be enough? We can even package this in a gadget for it to be more appealing. We can also accommodate for Cloud features like reliable backups in this p2p setting, a copy of one's photos can be backed up at another location, e.g. parents' house.

Concept for personal server, aka "Box", read more on fx.land

2. Doing both backend and frontend is hard

Absolutely! We need to fix this. Google and the like have armies of developers working on each app, how can anyone compete? Don't fret! We can seek help from millions of frontend JavaScript developers around the world! They are already building all kinds of cool apps. Backend is not really an issue, many of them are well versed at using serverless cloud functions. So if we can provide the same "Developer Experience", can we have these brilliant people aboard?

To address the need for backend, and also the limited compute power of the hardware that may be used, the solution that we're currently implementing is "Managed Services". For a start, we are developing 2 protocols on top of libp2p:

File Protocol: for uploading and downloading files, this provides the function of a traditional file server.
Data Protocol: for persisting JSON data. This would be the Web 3.0 equivalent of REST or GraphQL endpoints. By linking the JSON data as DAG-JSON and saving it with IPFS, we also get the conventional database-like queries.

Work on more managed services are on the way. We'll need Machine Learning services for an app like Photos soon!

These managed services are packaged in a JavaScript module, @functionland/graph, so the complexities of what happens behind the scene is abstracted away for the app developer. They'll have an experience very similar to using serverless functions. Voila!

This work is being pursued in the "box" monorepo:

github.com/functionland/box

3. If an app is open source, how can the developer make a living from it?

This one is a long standing question in open source, and Blockchain may finally give us an answer. It's ironic that prominent blockchains such as Bitcoin have become somewhat centralized, true decentralization happens when average people run validator nodes. That's not the case today, but if there is a future where every home has a server, true decentralization can start! Newer generation blockchains are way less resource intensive, take Mina for example, a snapshot of the whole chain will always be 22KB!

So we can have home servers also be blockchain validator nodes, effectively printing money around the clock! We can then have a decentralized version of an App Store, downloading and updating apps from this store can happen on-chain and be paid. But users won't be paying out of pocket, they'll pay with a portion of coins mined on their server (freedom tax :).

This aspect is still very much at ideation stage, we'd love to brainstorm and hear what you think about it.

4. Would the User Experience be compromised in any way?

If anything, UX will be improved. There is no privacy concern, users physically own their data. There's no advertisement, no lock-in. And in open source, anything that has use cases also has a community of contributors around it. So no longer we'll see useful apps become adware (anyone remember ShareIt?). No orphaned app stuck with last update from years ago. All the cool things that we developers enjoy in our open source workflow, now up for grabs by the masses in everyday apps!

Eagerly looking forward to hearing your comments! Please don't hesitate to bash the whole idea to the ground! We don't mind at all. Only in common wisdom shall we thrive, no one soul knows all, most definitely not the noobs who started this :) -- Love and peace, Functionland