Short retro for vibe-vendoring opentimestamps

[sir-opti]

Project goal

To test the concept of vibe-vendoring, which roughly means replacing third-party module dependencies with bespoke, self-maintained source code produced by coding LLMs, I had taken the opentimestamps npm package as a prime candidate. See #2854. The goal was to, within a reasonable timeframe (a few days), get to a point where the limited functional scope integrated with SN produces similar or better results with 100% bespoke code, zero new dependencies, and has full wire-compatibility with the protocol-side of the ots system.

Rationale for picking opentimestamps

• The npm package uses a http library (request) that has been discontinued for over 6 years.
• It's a complex package, because it has a custom wire format that is, especially when compared to any average javascript package, abstract, complex, hard to debug and not fault-tolerant, and it has quirks. This makes it a really good test subject, because it ought to trigger all the problems one can possibly run into.
• It's not actively maintained, and therefore a risky element in the supply chain

The setup

I've used an isolated, bespoke forgejo environment where I've imported the following source repos:

• stackernews/stacker.news - the target integration
• opentimestamps/javascript-opentimestamps - the npm package to replace
• opentimestamps/python-opentimestamps - as the ultimate authoritative implementation

The environment is wired into Claude Code for claude-4.6-opus, Codex for gpt-5.4-codex and during the project I added OpenCode for glm-5, to be able to get a "third opinion". Message dispatch goes through issues/PRs/comments and a Kanban for assignments and prioritizing.

The plan

The plan was to execute the following steps:

1. Deep analysis of the integration points in SN codebase (only 1) and the 3rd party code underneath -> half a day
2. Spec out the replacement code at high level -> half a day
3. For each item in the high level spec, make a detailed spec, compare it against the 3rd party implementation, and implement tests and actual code -> 1-2 days
4. Test everything through -> half a day
5. Integrate, end-to-end tests -> at most a day

What went well

• Deep analysis of the scope: what was implemented by SN, which codepaths are touched and which can be left out, what are immediate points of extension that may be needed in the future? This is something modern LLMs are really good at analyzing. The entire project did not experience any scope creep.
• Clear design spec: A plan was vibed up and cross-reviewed by agents. Though note that what it did was mostly just suggest stubs based off what it read in the javascript implementation and make sense of the interactions.
• Dependency elimination: initially all remaining third party dependencies were swapped by reusing an existing dependency (cross-fetch) and that too was later abandoned for built-in fetch, to get better test integration.

What didn't go so well

• Initial implementation. At first round, I found that too many shortcuts were being made in the detailed plans and implementation phase and inferior, buggy code was being delivered. I threw that away because trying to salvage it into a deeper level of detail made a lot of errors recur between fixes, like in the old days.
• On the second attempt, I had the bot fill out the structure as planned, and then iteratively work over that, realizing code for one testable flow at a time. Although this went okay-ish, it was slow, with still a lot of bugs, and a lot of mistakes were made that were hard for me to catch because I'd never written any ots code before. My own familiarity with the code was clearly a bottleneck.
• To solve the endless correction loops and my own lack of knowledge to steer the results properly up front, I decided to just dive in and code the bulk manually. This worked a lot better, but the exercise wasn't pure "vibe" at this point anymore. It also took less time.
• A lot of bugs were found by subsequent agent review and analysis, but 2 important things were missed:
  1. An encoding mistake in the wire format wasn't found despite 4 review rounds with different models and frameworks after it was introduced. The cause of this was a test reinforcing the failure.
  2. A function signature call error wasn't spotted by any agentic review (nor tests) in 2 rounds, which pointed to a context deficiency and tests were too shallow.
• Tests were generally less generously covering code than desirable and it took fallback to meat coding tests to actually get into a state where issues were caught.
• Whenever bots are focusing on fixing bugs, details like inline docs experiences quality decline.
• I expected to be done in a week at most, but the entire project ultimately took 2 weeks.

Remedies applied during the process

1. Improved review templates to be more concise, in multiple rounds, trying to make both process and context as explicit as possible. This helped most in the final rounds of analysis and review.
2. Context poisoning is still a thing. If a poorly analyzed finding sits inside a review conversation, bots keep returning to it. I circumvented this by directing reviews from issues instead of PR comments. This helped keeping the context clean.
3. After an LLM assisted round of documentation generation with tight human review of every line, and subsequent doc checks for each intermediate commit, keeping up the documentation quality became more reliable as a process. This didn't work as well for unit tests though.

Chicken & egg

The biggest problem observed was that it is hard to direct bots on building a codebase that you're not intricately familiar with, because it will be very hard to spot problems. The benefit of (pre-vibe) dependencies was that often someone went through the pain of understanding the domain and then coding the thing, before publishing. So there was domain knowledge that, although being external to the end user implementation, could make judgement and guide the quality of the implementation. LLMs do not get this domain knowledge unless you feed it to them. Patterns can of course be recognized and replicated, but unless the underlying domain knowledge was perfectly documented (rarely happens) the process is lossy when doing pure vibe-vendoring.

To remedy this, it seems important for the viber to gain some domain knowledge and understand many of the details, so that bots can be corrected when bad context gets filled in or assumed. This in turn implies that the process where most of the initial code was hand-written and then iterated over by bots is maybe not only the fastest in terms of lines per second, perhaps the fastest overall when taking into account knowledge debt. When combined with continuous documentation improvement and deep unit tests that spec behavior, it will be easier to vibe-maintain the resulting code, also because there is more knowledge.

TL;DR

There seems to be no magic bullet to go from zero-to-llm-hero in this for dependencies that need to be production quality, at least not without a steep learning curve for those that are accountable and want to review. Without human domain knowledge, the risk of bugs grows exponentially. Ultimately, this means that human time investment remains a real factor.

[huumn]

Thanks for the write up! It'll save me attempting to pulse the experience into my memory.

I didn't know you spent time manually coding it so I was a little more pleased with the experiment initially.

The benefit of (pre-vibe) dependencies was that often someone went through the pain of understanding the domain and then coding the thing, before publishing. So there was domain knowledge that, although being external to the end user implementation, could make judgement and guide the quality of the implementation. LLMs do not get this domain knowledge unless you feed it to them.

Without human domain knowledge, the risk of bugs grows exponentially. Ultimately, this means that human time investment remains a real factor.

Yep. My dep ecosystem conclusion: competition and quality of deps should increase at least, and frivolous deps can be vibe-vendored. Quality go up and trust go down, but there's no dep singularity just yet.

[sir-opti]

spent time manually coding it

Most importantly, the encoder/decoder baseline. Once that was in place, the rest was easier for the bots to do, except the special rules around op/attestation tags in timestamp serialization - I refused to code that manually and I paid for it in terms of time it took. For me the most (and ultimately most valuable) time went into manually deserializing a couple of ots binary files from hex to get a better mental map of what input/output looked like.

It could be valuable to formalize this skill ("here's a hex string, here's code, make sense of it, improve docs") to a bot. But that feels like a precision job that will take a lot of time to get right; something to slowly build out perhaps.