Website
I've made a lot of progress on hashpool in the last month. Over the holidays I worked on a website repository. I have a little experience hosting a website thanks to my BitDevs chapter but I knew I needed a more modern framework. My BitDevs site uses jekyll, which is not well supported anymore. I knew a lot of other BitDevs chapters were upgrading to Zola from the original jekyll repo so I went looking for a nice Zola theme to start with.
I found a cool looking theme called PaperMod and got to work importing all my dev logs there. This was a relatively easy task which was great for continuing my green box streak while keeping my work load low during the holidays.
Once I finished with the dev logs I turned my attention to molding the site layout to my needs. This was a much more difficult proposition given my unfamiliarity with the tools and concepts of web design. But I powered through my skill issues to get it in reasonable shape. I even managed to fix a display bug and upstream it. Nice!
You can see the site by cloning the repo and running it locally with zola serve. I have not published the site yet. I want to write some educational materials that will form a sort of landing page and clean up this part of the site layout so it looks good. I ran out of holiday time before I got that far. In the new year the kids returned to school and I had much more bandwidth to tackle harder coding problems so I turned my attention back to the Hashpool repo.
Full Proof Sets
My big Hashpool accomplishment this month was rewriting the network transport layer to carry an array of 64 pieces of data instead of one. This enables hashpool to generate a full proof set for each submitted share. Now each time a miner submits a mining share, they receive ehash tokens adding up to exactly the difficulty of the submitted share. With this capability, I think it's fair to say that ehash issuance has upgraded from a proof of concept to a featureful implementation.
In the process of building out this infrastructure I introduced the concepts of domain and wire types. A domain type represents the piece of data that you are working with, in this case signing keys, blinded secrets, and blinded signatures. A wire type is the encoded version of the domain type suitable for network transmission. My wire types carry the entire array of domain items.
I already had Sv2 structs that can each store a public key and some metadata fields in a way that the SRI encoding library is willing to encode. In order to transmit a full keyset I needed to store 64 keys, one for each power of 2 denomination that can fit into a 64 bit value. Essentially, to represent every possible denomination you need one key pair for each bit position. It's very unlikely that anyone will ever mine a share with 63 leading zero bits but it's technically possible and for a detail-oriented completionist like me it just feels right to build it this way.
Since SRI encodings are fixed length you need to have enough space to transmit an array of 64 pubkeys. Pubkeys are 33 bytes so that works out to 2112 bytes for the whole data structure, give or take a few header bytes. Unfortunately, the standard byte array types are not very granular, they jump from 255 bytes to 64,000 bytes. Oof! I have to use a 64kb data structure to store just over 2kb of data.
It would probably be a simple optimization to add a more fitting byte array type and stop wasting 62kb of space in each message but it's not a priority at this stage of development. This might be a good task to tackle when converting my fork of SRI into an extension.
I used the index of the array to represent the amount of the tokens that key is used to sign. So the signing key in the first array element is used to sign tokens of amount 1, and the second array index corresponds to amount 2, then 4, then 8, and so on. The same applies to the arrays of blinded messages and blind signatures.
I took the strategy of implementing the domain and wire types three distinct times for the three domain types. Then I upgraded the network messages to use these wire array types and put in shim functions to convert an array to a single item (or vice versa) by discarding all but the lowest index entry. Kind of like adding new lanes to the highway that are closed while still under construction.
With this infrastructure in place, I deleted the shim functions one at a time and implemented the necessary functionality to hook up the wire array types to each other. First I deleted the function that discarded signing keys and updated the proxy service to import all the signing keys into the cashu wallet struct. Then I used those keys to generate blinded messages when a share is found. Next I deleted the helper function that discards blinded messages and updated my cdk code to sign all non-zero blinded messages in the array and return a wire array of blind signatures. Lastly, I deleted the final two helper functions (I was surprised to find that one of my helper functions wasn't even used. Hah!) and imported all the signatures into the wallet to create a set of proofs totalling up to the work value of the mining share.
With this change, I could finally log out the complete proof set and verify that the tokens summed up to the work value of the share. A month of work finally realized! w00t! \o/
Here is the commit message:
ACHIEVEMENT UNLOCKED: mint full ehash proof sets
- pass the full premint secret collection to cdk for signing
- improve logging
- delete remaining functions that discard all but one proof
Refactoring
In the week since then I have been working hard to collapse the three redundant implementations into a common interface. I ran into a very frustrating snag with the Encodable and Decodable derive macros...again. I had a hell of a time deriving these traits on my structs the first time around. I expected to be able to just copy the existing structs and avoid additional trouble with these macros. Oh how naive I was...
It turns out that if you name a field in your struct data and try to derive Decodable (which is confusingly remapped to match the name of the serde version of that trait, Deserialize, even when you aren't using serde...idk why! ¯\_(ツ)_/¯) you get mysterious compile errors. The same mysterious compile errors you get when just about anything goes wrong with these derive traits.
I plan to rebase my fork against the latest SRI release and then open an issue with this feedback. I definitely want to leverage my experience to smooth the way for future SRI developers.
It took me a few days to work out this solution but just today I was finally able to combine the blinded secret and blind signature types under the new DomainItem/WireArray generic implementations. My last commit added 84 lines of code and removed 370. Damn it feels good to be a gangster.
Next Steps
This probably represents two thirds of the code to be deleted. I think I can also collapse the default impls for BlindedMessage and BlindSignature. I still need to unify the signing keys under the same wire and domain generic structs but I'll need to refactor the amount value out of Sv2SigningKey first.
Once this refactoring is completed to my satisfaction I need to clean up my cdk fork by rebasing it against latest master and getting ready to open a PR with my new functions. Again, I want to leverage my work to pave the way for future devs. My PR will add the BYO-network-transport use case to the cdk library. I can't wait to see what other stuff people build with this functionality.
My next big Hashpool development goal will be to stand up a HTTP mint service and wallet CLI. I want to be able to stack ehash tokens in my cashu wallet as I find mining shares in real time. I think I will get people really excited once they can see ehash tokens land in their wallet. I might even use my hashpool.dev domain to stand up a testnet instance. Eventually, I could even run a mainnet instance that lets people donate hashrate to the development fund. In return, they get worthless ehash tokens that can't be redeemed. I bet some folks would make that trade to support my work.
Delving Bitcoin and the Optech Podcast
Last week I contributed my first post on Delving Bitcoin. I discovered in the Optech Year in Review issue a discussion of representing mining shares as ecash tokens from May. I had seen the original post on the topic when Gary shared it with me at the time but I hadn't seen the discussion that followed. I didn't have a ton of answers back in May but by now I had plenty of new research to share, which is exactly what I did. You can read it here.
The following week the Optech team asked me to join them on their podcast. Hell yeah! I was stoked! I think it went really well. You can listen here.
NEMS and other travel
I'm travelling for the next two weeks, so I don't know how much deep dev work I'll get done. This week I am attending NEMS, the Nashville Energy and Mining Summit, to meet with mining industry folks and shill my project. I am most excited about the BitDevs before the conference. BitDevs is my jam and they seemed very interested when I offered to talk about my project.
The following week I'll be travelling on personal business, so I'll be looking for easy contributions to score a green check on those days. Now that I think about it, I need to update the screenshot on the README page...🤔