Bodybuilders commonly use a technique called "bulk and cut" to achieve their fitness goals. The bulk phase is characterized by a calorie surplus combined with intense resistance training to rapidly gain muscle mass. This comes at a cost: it introduces a lot of undesired body fat. During the cut phase, an athlete will switch to a calorie deficit and a more balanced set of resistance and cardio exercises to maintain muscle mass while reducing body fat. The overall effect is a phased march toward a more perfect physique.
I often return to this metaphor when crafting a development plan for my software projects. The concept of bulk and cut maps very well to the software engineering process of rapid feature development punctuated with code consolidation and retiring of technical debt. This quarter Hashpool has entered the cut phase. The results have been quite good! A phased march toward a more perfect codebase. 💪
Rising Tide
Hashpool builds on top of two foundational software projects: SRI and cdk. This choice allows Hashpool to benefit from the continual improvement in features and code quality in those two projects. When implemented well, this design philosophy creates a massive tailwind; Hashpool gets to surf the wave of bitcoin library development. Sound strategy is one thing, execution is another matter. Good execution means picking the right projects, maintaining those lib dependencies, and contributing back to add your displacement to the rising tide.
From the beginning the cashu project has been steadily adding features, improving privacy and security, and attracting talented builders. Thanks to the relentless work of the cashu team, progress seems to be accelerating. 🥜
The SRI project has been through a years-long grind toward a stable protocol library and usable application layer. It's been a slow process but in recent months SRI seems to have shifted into high gear. The protocol crates are rapidly stabilizing, the sv2-tp binary has shipped, and the introduction of the sv2-apps layer seems to have attracted a new class of builders.
I'm obviously biased but from my perspective all I see is flat glass and a clean swell. 🤙🏄
SRI 1.7.0 and sv2-tp
The first major undertaking of the cut phase was to upgrade to SRI 1.7.0 cargo libs and delete most of the vendored source code from that project. This also meant moving off of Sjors' ancient sv2 template provider bitcoin core fork and onto the latest core release paired with the new sv2-tp sidecar process. This is a continuation of the work I began with the 1.5.0 upgrade: migrating cashu quote and ehash logic to their own dedicated crates. That work cleaned up the bulk of the business logic but there were still some lingering Hashpool artifacts in the vendored SRI libraries. The most satisfying results from this sprint were updating the cargo libs and deleting over 14,000 lines of vendored code. Talk about cutting the fat!
Redeploying the Test Pool
No plan survives first contact with the enemy. I needed a way to expose the newly refactored codebase to the enemy (production) so I rebooted the hosted testnet pool. In the process, I upgraded and refactored the deployment scripts into two scripts. One script assumes a nixOS environment and builds the binaries in place before shipping them to the vps. This was my original plan, which leverages the deterministic build guarantees of nix. Unfortunately, I encountered significant difficulty in standing up a private, anonymous hosted nixOS vps. I might have to wait and see if the landscape improves in this regard. (If you know of a good provider that accepts bitcoin, doesn't KYC, and offers nixOS out of the box please let me know!)
I preserved this option in the build script but shifted to a more conventional Debian vps architecture. This script ships the source code to the vps and builds the rust binaries in place. This ensures the binaries are compatible with the host architecture but it requires significant host resources as part of the deployment process. I find that I need to regularly clean the build artifacts to avoid filling the hard drive on the host machine. This is a double edged sword because the first build after you clean takes FOREVER. #rustlife
In the process of upgrading the deploy scripts I found it necessary to clean up and clarify the target hashrate config options. It has been a long standing frustration for me to keep both the CPU miner and external Bitaxe miners both working in the codebase at the same time. I like to use a CPU miner during development to avoid the need for external hardware and configuring network access to the dev environment. But any real pool deployment needs to use ASIC miners to meet the network difficulty target. In the past, whenever I fixed the problems with one kind of miner it broke the other kind. So I took the time to uncover several bugs, misconfigurations, and misunderstandings of how the config settings work. With these improvements I can use the CPU miner for dev work and keep the testnet deployment humming along with Bitaxe test miners.
With the testnet pool back up and running I also added a config-only deploy option and fixed several bugs that surfaced on the hosted deployment. Uncovering these problems is exactly why the testpool is so crucial to the development process!
Eliminate CDK Fork
Another longstanding frustration of mine has been the need to run a custom fork of cdk to support ehash. Maintaining a custom fork is best avoided because it's a significant drag on development so my next sprint goal was to finally remove the need for this fork. I previously opened a PR to the NUTS repo to add mining shares as a payment method. The PR was rejected but it exposed the necessary fields and capabilities to support new payment methods without requiring protocol changes. Now it was time to leverage those fields. I extracted the custom payment processing logic into a new cdk-ehash library and updated my cargo dependencies to the cdk main branch.
In the process of getting on the latest cdk release, I also updated hashpool to use batched minting (which the cashu team shipped in my absence). Squad goals. \o/
Prometheus and Sv2 Apps
My greatest frustration with the original test pool deployment was my vibe-coded stats implementation. I spent far too much time trying and failing to fix the many hashrate graph bugs. After upgrading the SRI protocol crates, I thought I should take another crack at this problem. I had been hearing about the new Prometheus integration so my first goal was to drop my vibe coded AI slop stat service in favor of a proper production solution. This was a great success! I not only removed 4,000 lines of code but finally fixed the bug causing reported hashrate to drop to zero and stay there for 30 minutes after a deployment.
One of the miners on my testpool deployment would only stay connected for an hour at max. This indicates a connection thrashing problem. I fixed the bug on a git branch and then discovered that it had already been fixed in the sv2-apps translator process. So I discarded that branch and upgraded my old translator role-based architecture to use the new apps-based architecture. Unfortunately this meant vendoring the sv2-apps source library, which added 13,000 lines of code to the repo. I guess this is the start of the next bulk phase. ¯\_(ツ)_/¯
Sv2 Apps Cleanup and Upstream Contributions
Obviously, I don't want to keep all this vendored sv2-apps code any more than I wanted to keep the vendored protocol code. So I created a plan to remove the newly vendored sv2-apps crates. It mirrors the cdk migration effort: extract the custom logic into a dedicated repo/crate, open PRs to add the necessary fields and extension hooks to the upstream libraries, and delete the vendored code once the new hooks are added to an upstream release.
I have identified three PR candidates and opened the first PR. I also discovered a need for a Sv2 protocol update.
Heatpunk Summit and Next Steps
In February I attended the Heatpunk Summit and sat on a panel discussing pool payouts. The summit was excellent. It was refreshing to attend a bitcoin conference populated with real world builders: plumbers, electricians, mechanical engineers, hydronic engineers, and hot tub salesmen. These people are the motive force of the real economy and I believe they are key to taking bitcoin mainstream. Real transformational change happens from the bottom up so it was nice to focus on these actors and their problems instead of memes and media talking points.
One of the main topics of the panel was investigating ark for pool payouts. The more I think about how to fund lightning channels from block rewards, the more convinced I am that it is the wrong solution. I feel a strong pull to begin exploring ark as a payout mechanism but my inner engineer knows that the project needs to be more stable and maintainable before embarking on this project. A few more bulk and cut phases might get Hashpool exactly where it needs to be. I can't wait to get there.