2025: Year of the Linux Desktop Laptop
I migrated to a new linux dev machine. I have bounced between linux and Mac many times in the past. Previously, when I went back to Mac the deciding factor was poor linux support for laptop hardware and corporate software I was required to run. Last year I quit the corporate world forever, so the balance has shifted.
It seems like linux hardware support gets better every year while Mac hardware and (most infuriating) OS level software restrictions continue to enshittify year after year. It feels good to break free. GFY Apple. 🤓🖕🍏
Build Issues
When I tried to run Hashpool on linux for the first time the bitcoind process failed to build. After troubleshooting I realized that I was referencing an old commit on sjors' bitcoind branch that has since been orphaned by a force push. The build cache has been insulating me from build failures all these months. A CI system would catch these failures, but it's not a problem unless you have a team of people working on the same codebase. Squad goals.
I updated my bitcoind branch to the latest revision and got Hashpool running again. When you encounter a build issue that's likely to reoccur it's always a good idea to try and find a permanent fix. So I turned my attention to the build system. At the beginning of this process I had a very limited understanding of nix, devenv, and nix flakes. But I knew that flakes are good for locking down your dependencies so I forged onward in an attempt to do exactly that.
I spent a few days vibe coding a nix flake. After much struggle and frustration, I actually got it to build. Unfortunately, I quickly realized that this was not a solution to my problem. As for vibe coding a nix flake, I do not recommend this path to other developers. 😅 Probably better to stick to vibe coding the frontend.
I can't use a nix flake to track sjors' branch because his repo is not a flake itself. Sjors would need to commit a flake.nix file to his repo, and keep it up to date. In light of this new information I opted to manually keep track of the bitcoind revision and write a just recipe to assist with the process of updating it.
When I was satisfied with the new solution I deleted the flake file. I will definitely be using flakes in the future so it's only a matter of time before I bring flake.nix back but I will wait until the need arises.
Logging
When I was troubleshooting the linux build I opened a PR to the SRI repo with my log errors. If you look at the issue you can see I only included screenshots of my logs. This is because devenv processes only output to the console. I don't know if or how you can copy text out of the devenv up terminal environment so screenshots were all I had. It was good enough to get the job done in this case but far from optimal. After this experience I decided that logging was a top priority.
I updated my devenv.nix file to get output logs working for all processes. These logs will be essential to troubleshoot issues in the future, guaranteed.
Version 0.1
With a new computer and new, hard won build expertise I was ready to refactor. One of the first things I wanted to do was to rip out the proof generation from Hashpool. The new quote design completely removes the need to handle this function in the SRI code.
I tagged Hashpool v0.1 to keep a reference to the first working ehash implementation. I had a feeling I would want to refer to this code in the future and the change in development direction seemed like a good reason to drop a tag. I didn't tell anybody because it was kind of a nothingburger development. But, to my surprise, Bitcoin Optech picked it up! 😳 Do you ever feel watched?
I have never run an open source project before so I don't know if or what proper versioning etiquette is. This release has no build artifacts, no instructions, no release notes, and thanks to the lack of a nix flake it probably won't work at all very soon. Fortunately, none of these things is a problem if you don't have any developers or users. But hey, the first version was tagged so let's party! 🎉
Mint Quotes
When I finally started working on quotes (a few weeks after identifying the need) I found it simple and straightforward to add a new quote type to my cdk fork, and in a hashpool commit replace the proof generation with quote generation and return the quote_id.
Cool cool cool. So it runs and logs a quote id, but how do I test the quote system? The idea is to have the user wallet poll the mint and retrieve tokens when the quote status is updated to PAID. How do I do this with the current design?
Well, I can't really. The mint is just some cdk code hacked in to the pool role. It doesn't have an http api (at least I hope not!) because SRI has its own whole communication protocol. Bolting on a cashu mint would muddy the waters and explode the complexity of building and supporting this system.
Running a separate process is a much better design because it separates the Sv2 and cashu protocols with a minimal interface between them. This way, if the mint has a bug it won't interfere with mining operations and if the pool process has an issue it won't interfere with ecash operations.
Also, decoupling these processes will enable operators to deploy and scale them as needed. The load profile of mining and ecash operations will differ. They will experience load spikes at different times and with different magnitudes. Plus, developers can specialize around the different roles. Personally, I expect to do more work in the mint than the mining side of the stack.
Mint Service
I stood up a standalone mint service as a devenv process. This was more difficult than I anticipated due to dependency issues between cdk and SRI. I eventually found that by using nightly rust I could enforce a consistent rust version between SRI and cdk. This was a quick and dirty solution that let me move forward but I think it highlights the need for a nix flake.
I understand the 'right' way to force a rust version is to use a nix flake. I was able to find a workaround but nightly rust is unstable and will eventually come back to haunt me. This makes me wonder when is the right time to bring the flake back?
I haven't updated my SRI code in quite a while. I need to check if more recent SRI and/or cdk releases can resolve some of the dependency issues I encountered. I am hesitant to update SRI while the ecash code is in flux. And I don't want to slow Hashpool development any further to spend more time on the build system. I could have the best damn build system in the world but what good is it if I still don't have a working mining pool?
In the back of my mind I am questioning what the best order of operations will be. My overarching goal is to sprint to a working implementation of an eHash mining pool. When issuance is working I will migrate to a nix flake and iron out my dependencies. With flake in hand, I can update to the latest SRI release and pin that version. We'll see how accurate this plan is in the coming months.
cdk CLIs
A big part of my struggle with the build system was trying to get the cdk cli binaries to build as part of the development environment. I thrashed around for a while on this problem and finally just committed the binaries directly. This is not ideal because it adds an extra step every time you update the cdk code. If that update breaks either CLI binary you need to recompile it and re-commit. So it's an infrequent update that is usually not required. A recipe for breakage.
Again, I prioritized expediency. This isn't a problem worth spending a lot of time on and it can be resolved when I nail down my dependencies with a nix flake. In my engineering career I have found that quick little hacks like this either have a very short lifespan or they last a shockingly long time. There aren't really a lot of hacks that last a medium length of time. I have a feeling this is one of the latter cases because it's so darn easy to just copy a new binary over and commit it.
With the CLIs ready to go I can now query the mint process for quotes to test that functionality when it's ready.
Next Step: Redis Queue
At first I intended to build a gRPC messaging layer between the pool and mint processes. When I started planning this I realized once again that I would have to reconcile the different async models of SRI and cdk. SRI uses the MPSC model and CDK uses tokio for async operations. I'm not really familiar with these models or how to reconcile them between processes but it looks like tokio has an MPSC channel for this purpose.
I don't think I need to go there, though. I will also need to queue requests at the mint to protect against a load spike. Pooled mining is a very time sensitive operation. Milliseconds count, so the last thing I want to do is slow down the mining services to wait on the less time sensitive ecash operations.
When I thought about it for a bit I realized that none of the mint operations need to happen synchronously. This means I can skip the gRPC messaging layer entirely and rely on redis for all communications. I only need two message types: create quote, called when a mining share is accepted; and update quote, called when the pool is ready for the mint to release the ecash tokens. In Hashpool, update quote events will be fired when the share and block template are both validated.
In Axepool, update quote will be fired when the pool payout lands and the user is able to redeem their sats from the mint. The fact that the same architecture works equally well for two totally different pool designs is a big hint that we have arrived at the correct abstraction. Big props to team cashu for figuring out the hard stuff and capturing it in a specification! Y'all are the goats of NUTs (not to be confused with the goats' nuts).
Wait...did you say Axepool? WTF is Axepool?
Axepool
One of the goals of Hashpool from the beginning was to enable a proxy pool. I do not mean this term in the same way that people refer to a small pool mining on the same template as a larger pool. When I say proxy pool I am referring to a mining pool built on a stratum proxy server.
This is a pool that mines upstream to another pool. It's not a widely understood concept yet because no one has built one before but I believe this is the future of mining. I plan to talk about this idea at length at btc++ Austin in May of this year. Check out my talk to learn more of the theory behind this idea.
Hashpool today is not a proxy pool. When discussing these ideas with friends, I explained how the only realistic path to build a proxy pool today is with OCEAN because they are the only PPLNS pool left standing. All we need to build it is an Sv1 proxy with the right callback hooks to perform ecash operations. After this conversation, my pal Johnny decided to go out and build it.
What a chad! Johnny gets it. We must accelerate. He's developing a minimal Sv1 proxy with the goal of enabling ecash mining pool withdrawals as soon as possible.
It's called Axepool because we are targeting the Bitaxe market. Bitaxe runners today mostly solo mine because the user experience of existing pools is terrible for micro miners. Axepool's primary goal is to accelerate the adoption of pleb mining by solving this UX challenge. We're going to skip over eHash and build the design described here which enables users to mine privately and withdraw to an ecash wallet.
A big benefit of this design is that redemptions are super simple. Instead of rotating the ecash epoch on a clock time schedule in order to enforce PPLNS share accounting, we can just have one ecash epoch per OCEAN block. Axepool can simply sum all shares received in between OCEAN blocks and pay them out as soon as the payout hits the Axepool lightning node. All we need is a BOLT12 callback to trigger the quote update message.
The reason this is possible is that OCEAN implements TIDES (modified PPLNS) at the pool level so we don't need to reimplement it again at the proxy level. We can piggy back on TIDES to smooth out the Axepool payout structure across blocks.
Axepool development will happen in parallel with Hashpool. It will reuse the quote based ecash design and have a drastically simplified payout system with no intermediate eHash asset. Users will submit mining shares and wait for the next OCEAN block to sweep their ecash tokens from the mint.
HRF Grant
HRF has awarded me a grant for Hashpool development! Holla!
Someone decided to pay me to write open source code. That means I am now a professional open source developer. This should leave you feeling very confident about the future of bitcoin mining.
