What is Open-World Compute?
Serverless computing has been promised as the next big thing™ in computing. There are many advantages: not needing to build and manage complex backend code, and granular billing per invocation means that you only pay for what you use. Many of these systems have adopted workflow systems (e.g., AWS Step Functions, GitHub Actions) on top of batch requests. Unfortunately, the systems to date have serious limitations: they’re locked into cumbersome proprietary APIs, have a finite number of integrations, and often don’t handle partial failure elegantly (e.g., can leave a database in an inconsistent state if something goes wrong in another step). Furthermore, many of the compute offerings today lock developers into an ecosystem, making it difficult to switch to other solutions in the future.
A more principled generation of workflow systems has developed to address some of these problems. They typically depend on the developer writing their code “correctly” and not doing things that make their code repeat work on multiple calls. This doesn’t play well with many common functionalities: what about when developers want to access time or random numbers? They also often don’t handle rollback out of the box. In short: they’re closed systems that depend on the programmer working hard to ensure that abstract properties are respected. This is a real problem: networks can (and do!) fail. Developers write buggy code (we all do it). It’s an extremely high bar.
IPVM, or the InterPlanetary Virtual Machine, brings content-addressed computation to IPFS using Wasm, PKI, and capabilities. IPVM liberates computation from its dependence on prenegotiated services and paves the way for interoperability.
IPVM (and Homestar) takes a different approach. Instead of depending on the developer to be careful, it restricts their programming environment. For example, WebAssembly (Wasm) can be limited to deterministic computation: any code that you compile to Wasm has to have these properties that we want by definition. IPVM also acknowledges that failure happens and pulls from recent research about hiding that as much as possible by leveraging content addressing and Wasm.
Fission's vision for open-world compute is highly usable and collaborative. We want an open, multitier system that runs everywhere, connects all the world's users and services, and is substantially better than Web 2.0. Anyone will be able to broadcast a task type on the network, and it will be automatically usable without having to fork the project or ask permission.
There are several key characteristics we're considering as we build IPVM. As mentioned earlier, idempotence and determinism are key to reliable computing, but they can be challenging to understand and implement.
Wasm allows developers to bring their own language, run computations on a server, in their browser, or even on their phone, and run them in a deterministic environment so that the same inputs always produce the same outputs.
Furthermore, when we run computations with CIDs, the results output more CIDs. The CIDs do not get recomputed and reproduced each time we run them - they are essentially cached in IPFS. When someone applies that same input, it simply calls back that cached result. No new records are being created, making it idempotent. This comes for free on Merklized data, but we can also Merklize data that's read from anywhere and cache it for later use.
While pure computation is useful for many applications, you still need mutability in many cases. There are queries (reads) and pure computations, and these can go back and forth as long as we end with mutation. Because we are not giving users the ability to define the order of execution, we can reorder computation so that the mutations happen at the end.
If we bring these three properties together - determinism, idempotency, and purity, we can achieve virtual resiliency. This gives the developer peace of mind, knowing that, eventually, the computations will work no matter what crashes or curveballs are thrown at the system.
We also want replayability. The cached computations, or receipts, never have to be re-run. And when we need to make non-deterministic requests or work with time of day or random, instead of losing out on the guarantees of deterministic compute and having to use client libraries, the node can call itself, store the information, and run the RPC call. Then if we run it again, we can have replayability because those previous computations were already saved. This creates a "run once, never run again" system, making it more efficient and scalable. Using capabilities based on UCAN, we can determine who can run different computations. Perhaps one job is better for Bacalhau, another for Fastly, etc. Maybe you only want computation to happen on a cluster of machines that you and your friends run and no others. We let IPVM handle the matchmaking and authorization checking so that different systems can work together to complete the job.
And the best part is this is all happening behind the scenes! The developer doesn't need to set any of this up. 👍
Compute over Data Summit 2023 Presentation
Fission Staff Research Engineer Zeeshan Lakhani gave a presentation on IPVM and his work on the Homestar implementation at the Compute Over Data Summit. When the recording becomes available, we will post it here.
Learn more or get involved by joining the IPVM working group.