hckrnws

Ruby is nice, but if you were to actually solve this problem for Python you'd print money. The deluge of knowledge required to actually navigate Python dependency management across all major OS's for the most complex use cases (typically ML stuff with deep compiler/tool chain deps) is nothing short of a nightmare. Especially on Windows, the rage I feel towards MSVC knows no bounds. venv, pip, (mini)conda, mamba, poetry, apt, yum... Obviously the last two are good to know anyways, but ugh the hours I've lost and will never get back to the rest.

Anyways, god speed my dudes.

(founder here). The interaction between high level language package managers and system binaries or compiled extensions is definitely on our roadmap. For example the ruby net-ssh gem relies on specific versions of openssl being compiled on the machine. Do you think this is particularly difficult in ML?

Is your frustration mostly from needing compile all this non-python stuff across environments?

Yeah, you're right on the money. This is basically insane in ML, but _especially_ in corporate environments, think F500, where sudo is disabled in containers and experimentation (in terms of trying different package versions) is quite challenging due to security restrictions.

Frankly, it's one of the reasons that we're seeing a lot of excitement around stuff written in Rust, but with Python bindings[1][2]. Cargo makes it so easy to build across various platforms and then just produce statically compiled platform specific python wheels. If PyTorch and TF had been written using Rust, any other other ultra high perf language with a tool-chain as reliable as cargo, a lot of folks lives would be easier.

[1]: https://www.getdaft.io/

[2]: https://www.pola.rs/

Hmm, the Debian package of ruby-net-ssh doesn't even depend on OpenSSL at all.

Interesting. Should it? I don't have much experience installing ruby gems with the OS package managers, I've always done it through the language-specific ones like bundler / gem. Here's a github issue showing the kind of things that come up when the versions are mismatched: https://github.com/net-ssh/net-ssh/issues/843

I was probably looking at an older or newer version to you, likely it switched from OpenSSL to pure-Ruby implementations of the crypto stuff.

I don't think you'd print money just by telling people if a dependency is hard or easy to install though. You'd need to actually solve the problem of getting the dependency installed or suggesting some alternative solution that gets the code written.

E.g. if I'm banging my head against a wall trying to upgrade to the newest version of PyTorch for some graph embeddings feature and everything breaks, knowing it's high risk doesn't help me. I still need to get the thing installed so I can use the feature.

Okay yeah, totally agree. I already know it's hard to install. If they gave me a path they were confident would work in terms of deps/versions to upgrade to, that I would pay for if it worked well.

Do you work at a company forcing you to develop on Windows? You couldn't pay to do to that.

I don't think the fundamental thesis makes sense. The most time consuming part of package upgrades is most definitely coding. When a dependency gets a CVE it isn't an optional upgrade we can do somewhere down the line when it's convenient. We have to do it by the next scheduled release, or worse, before the next scheduled release. Regardless of the package, there's always changes, some of them breaking, which take considerable effort not only to implement and test, but also to identify in the first place. That's where the difficulty lies, not in planning when to upgrade what.

I generally agree with you, but I'd caveat that a critical CVE is easier to take if you're on the latest version of that dependency (so you can easily apply just the patch and not other changes). We want to help you weave package upgrades into ongoing maintenance rotations so that this is the case.

You mentioned that implementing an upgrade is difficult because of the actual code change but also the testing and identifying what the changes are in the first place. We're starting trying to do a really good job of the last one - what are the changes in this next version, are they breaking, do they break for your codebase, and what should you do if they are? We want to move from there to automating as many of the code changes as we can, but my sense is the biggest value is in giving you the confidence that we at least captured all of the changes you need to be concerned about.

I clicked into this because I was excited, but everything feels like it's missing the mark for me. And, that pricing. Woof. That's steep for being a fancy changelog interpreter.

----

My biggest pain isn't with the plan. It's with the actual upgrade process. Things _always_ break. Every single time we do a major upgrade, there's this long tail of things we need to fix. Even worse is the undocumented changes that break. This gives me nightmares.

The problem with your product is four parts:

* It needs to solve much more of the problem. It's not really valuable to know that I need to change stuff. I can figure that out by running my package managers update tool and seeing what breaks.

* When things break, it's unbelievably time consuming to fix them. Even worse is the person tasked with doing the upgrade is likely not the subject matter expert on broken code.

* I need to know what's broken that wasn't documented. This is by far the hardest part of our upgrade cycles. It keeps me up at night knowing a major upgrade could break things that aren't covered by unit tests.

* Because dependency upgrades are so hard, we tend to do them in large swoops. We set aside a week or two of pain and power through them. Given you're charging $60/m, this suggests:

  * Your model is more aligned with small, continuous upgrades. Once you're "close" to the most recent version, it tends to be easier to upgrade.

  * Your model needs a better way to "gain pace" with the latest releases. If I can incrementally upgrade one package on every PR until I'm up to date, that'd be amazing.

I think this tool would be extremely value to me (likely $60/m/repo) if it:

* It ran the repo's test suite against progressively newer versions of dependencies. Showing when and where unit tests fail.

* Provide some means of "intelligent E2E" testing. Fire up our API/application. Tell us what changes when dependencies are upgraded.

* Fan out broken code to the subject matter expert. If an upgrade from X to Y is broken, look at the `git blame` to figure out who knows how to fix the code. Ping them asking them for help.

Thanks for the feedback. We know we're tackling a big problem here. Our ultimate goal is that if you use Infield we keep you on the latest version of every dependency, automatically. The road to get there can go a number of ways.

> My biggest pain isn't with the plan. It's with the actual upgrade process. Things _always_ break. Every single time we do a major upgrade, there's this long tail of things we need to fix. Even worse is the undocumented changes that break. This gives me nightmares.

Planning is one way to make those major upgrades go wrong less often. Our idea is that we can break large upgrades down into incremental changes that are individually safe and mixed in to your regular dev cycles, so that when you go to do the large upgrade you're not making such a big change all at once. This means things like fixing every deprecation ahead of time and making sure all the other dependencies you use are compatible with the version you're upgrading to.

We're calling this "upgrade paths" in our tool. Something like a major rails version upgrade is terrifying because of all the moving parts. Often you may have a dozen other gems that need to be upgraded in order to upgrade Rails. You said "If I can incrementally upgrade one package on every PR until I'm up to date, that'd be amazing.". We want to make that the flow for major framework upgrades, so you're merging in incremental improvements in the blocking dependencies ahead of time.

Undocumented breakages are for sure on our roadmap. We have two types in mind today: - Changes that are missing from the changelog - Incompatibilities across gems (for instance there was a time recently where if you had both datadog and newrelic installed and used elsaticsearch upgrading the newrelic gem would break prod)

We have some ideas on how to figure this out automatically (like reading code diffs with GPT rather than just changelogs), but the best way is to see upgrades out in the wild. As we see more and more upgrade experiences from our customers we'll be able to catch these issues and build this dataset.

I'd love to hear other ideas for making major framework upgrades safer.

> * It ran the repo's test suite against progressively newer versions of dependencies. Showing when and where unit tests fail.

Would this be something like `git bisect` for upgrades? We run your CI to figure out where you can upgrade to without something breaking? We're trying to figure this out statically by reading changelogs and building up a database of breaking changes. Our roadmap is to make this database better over time by pulling in more and more undocumented changes (by seeing customer upgrade experiences and by sourcing information from places like github issues). In my experience it's more common (and much more dangerous) that things break when your dependency changes in a way that wouldn't be caught by CI.

How are you computing upgrade paths? This seems impossible to do accurately, especially for Ruby, since you can't simulate the user's build's dependency resolution given that Gemfiles are dynamic.

Can you expand a little? Here's some technical background on what we're doing:

We have our own database of every version of every rubygems package alongside its runtime dependencies (like you see at https://rubygems.org/gems/pundit).

Then we parse your Gemfile and Gemfile.lock. We use the Gemfile to figure out gem group and pinned requirements (we run turn your Gemfile into a ruby AST since Gemfiles can be arbitrary ruby code; we use bundler's APIs to parse your Gemfile.lock). This gives us all of the dependencies your rely on.

Then we let you choose one or more package that you want to upgrade and the version you want to target (let's say Rails 7.0.4.3).

Now we have [your dependencies and their current versions], [target rails version], [all of the runtime dependency constraints of these gems]. We run this through a dependency resolution algorithm (pubgrub). If it resolves then you're good to upgrade to that version of Rails without changing anything.

If this fails to resolve, it's because one or more of your current dependencies has a runtime restriction on rails (or another indirect gem being pulled in by the new rails version). This is where the optimization part comes in. The problem becomes "what is the optimal set of versions of all your dependencies that would resolve with the next version of Rails". Currently we solve for this set trying to optimize for the fewest upgrades. As our dataset of breaking changes gets better we'll change that to optimizing for the "lowest effort".

Happy to elaborate.

Sure. There are a couple of stumbling blocks here:

> We use the Gemfile to figure out gem group and pinned requirements (we run turn your Gemfile into a ruby AST since Gemfiles can be arbitrary ruby code [...]

Here's an example Gemfile:

  if RUBY_VERSION < "3"
    gem "minitest", ">= 5.15.0", "< 5.16"
  else
    gem "minitest", ">= 5.15.0"
  end

This makes it impossible to statically determine the direct dependency requirements of a project.

> Now we have [your dependencies and their current versions], [target rails version], [all of the runtime dependency constraints of these gems]. We run this through a dependency resolution algorithm (pubgrub). If it resolves then you're good to upgrade to that version of Rails without changing anything. > > If this fails to resolve, it's because one or more of your current dependencies has a runtime restriction on rails (or another indirect gem being pulled in by the new rails version).

This actually isn't that big of a problem for Bundler, which uses pubgrub, which to my understanding is deterministic. A deterministic algorithm means you can actually take requirements and simulate builds. There are two places where I would be hesitant:

1. This determinism only works if you also know what the universe of possible dependencies looks like. In many corporate environments, this is not true! Many corporate environments use private registries that mirror public dependencies, and may have a set of dependencies available that looks different from the public registry, which means your simulated builds will resolve to incorrect dependency versions.

2. As you move to support other languages, many other tools use non-deterministic dependency resolution algorithms. In particular, NPM is famously non-deterministic, which makes it impossible to simulate an NPM build.

---

When you're trying to just determine that there exists a build that resolves properly, these issues aren't particularly painful. At $DAYJOB (and what I suspect you will want in the future), we are often trying to predict the exact build of a user given a new set of dependency requirements (e.g. so we can predict their new vulnerabilities), which means doing accurate build simulations is much more important.

This is just an excellent comment, thank you.

One note - we're not as concerned with predicting exactly the user's build as I think your $DAYJOB might be. We need to scan the space of possible valid resolutions that'd do a big framework upgrade, then pick an optimal (least effort) option from that space and chart your path towards it. Concretely that'll mean opening incremental PRs against your codebase where we provide lockfiles that are individually valid and accumulate in getting your upgrade done.

> where we provide lockfiles that are individually valid

Providing lockfiles is a really interesting idea! That certainly solves the "we need your non-deterministic build tool to reproduce an exact build that we found" problem.

We haven't explored this route yet because a lot of our customers use tools that don't support lockfiles (e.g. Maven - Java in general has a lot of legacy stuff).

If you want to build off of our work, our dependency analysis bit is open source: https://github.com/fossas/fossa-cli

Maven POM dependencies act as version pins, until you run versions:resolve-ranges to bump them. The only exceptions would be using a SNAPSHOT version (where each build gets a timestamp and is only cached briefly) or your Maven repo admin replaces the contents of a pre-existing artifact (and you'll see a lot of cached checksum warnings about it).

Very cool! Is the LLM-on-the-backend a recent addition or have y'all been thinking about that potential since 2020? Would be curious to hear a bit more about your journey to market

Thanks! We actually pivoted to working on this problem at the beginning of last year. We were previously called Syndetic and were building software for data providers. As we considered pivoting I spent some time thinking about the biggest problems I've faced maintaining software and started doing personal consulting. I upgraded a bunch of Rails and React apps for various companies over the course of 2022.

Toward the middle of last year I re-connected with Andy (our third cofounder) who I went to school with and have wanted to work on a company with for a long time. We did a bunch of customer discovery / product work to figure out how to take what I learned doing this by hand and turn it into a software product. That was exciting enough that we decided to bring Andy on as a third co-founder and pivot our YC company.

Allison and my background is in building data businesses. Before Infield and Syndetic we worked at a startup in the beverage industry where we standardized inventory data for every alcohol product sold in the US. As we got into building Infield we didn't expect to use LLMs at all. We imagined a similar human-in-the-loop expert system to what we've built before.

I've been extremely impressed with recent language model's ability to handle unstructured changelog text. For example, consider the following snippet of a changelog:

  Security:
  - Address an issue with password validation

  Breaking change:
  - The `foo?` method now returns a boolean instead of int

Very cool, I love hearing founder journeys. Best of luck to you all.

Congrats on the launch!

The dashboard looks really well done, especially the path identification.

I'm curious why you decided to start with Ruby. Anecdotally, it doesn't seem as big of a problem as other languages with either (a) more upgrades or (b) worse package managers.

Disclosure: I'm the cofounder of https://grit.io which also does some dependency management, but focused on automating the actual code changes.

Thanks! We wanted to start with a language that's relatively consolidated around a framework before expanding to more balkanized ones like JS.

Sounds a bit like the Renovate bot https://docs.renovatebot.com/, but with LLM changelog parsing.

I started using Renovate on a few project recently and I am impressed. Not quite yet to the point of letting it merge dependency updates for me, but for unearthing them and creating a pull request (after which my tests run automatically) is already a big help.

Andy here (Product at Infield). I've also been impressed with what I've seen from Renovate - particularly on the configuration side. How has it worked for you on major / breaking upgrades?

How does this differ from Github Dependabot? Here's an example of vulnerability it found recently: https://github.com/advisories/GHSA-hj3f-6gcp-jg8j. And this is free.

Dependabot lets you know if a dependency you use has a vulnerability and opens a PR. The problem comes when teams are unwilling to merge those PRs even if tests pass. In my experience there's a very binary split here. Some codebases (especially open source libraries themselves) will immediately merge a dependabot PR if tests are passing. Many companies though find these PRs languish because they really need manual review to see if the change is safe and then an engineer needs to budget time in the case where it's not. These build up.

Maybe it's just me but this wording seems weird to me, not sure why exactly:

> World-class Engineering teams use Infield to make dependency maintenance time twice as productive.

Specifically, "to make dependency maintenance time twice as productive." Feel like it ought to say "to cut the time they spend maintaining dependencies in half". I know that's wordier and maybe that's not quite it, but the wording as is feels a bit tortured to me.

Andy (Infield co-founder) here - agreed that wording was a bit clunky. Changed now. Appreciate the feedback!

No problem. Congrats on the launch!

What I would love to be standard in all product launches like this would be to have a mailing list per integration or, as with this, programming language.

I want to be able to drop my email address and only hear back when support for $MY_FAVE_LANG is available.

It's a bit buried, but if you go to https://app.infield.ai/hn and click "I don't use Ruby" we'll ask for your email/language and put you on our list.

Cool idea! I thought about it too a few years ago while doing some upgrades. Do you support more languages? And do you have it in CLI form?

Only Ruby right now. We want to make it work really well for one language before expanding. Which language would you want it in?

We have a command line tool you can use to send your lockfile to our server so we can process it, but you have to use our web app to view the results.

Interesting, might try it out!

How do you plan to solve the diamond dependency problem?

Adding Go later?

Definitely. I'm very interested in govulncheck and building tooling into the ecosystem to make dependency use safer.