TL;DR: Contributing to open source can be frustrating as the consensus needed for code to align to the project vision is often out of scope for many companies. This post dives deep into the obstacles and wins of two contributors from different companies working together to add the same proprietary connector. It's both inspiring and carries many lessons to bring along as you venture into open source to gain the pearls and avoid the perils.
We’re seeing open source usher in a challenge to the economic model where the success metric is increasing the commonwealth of economic capital. This acceleration comes from playing positive-sum games with friends online and avoiding limiting a community to a vision that only benefits a small number of corporations or individuals. It’s hard to imagine how to embed such frameworks within our current zero-sum winner-takes-all economic system. There’s certainly no shortage of heated debates around how to construct a harmonious relationship between the open source community and companies participating in them. Something we don’t talk about enough are the positive examples of when a coordinated effort in open source sticks the landing, and so many benefit from it.
This post highlights the extraordinary contributions of Erik Anderson, Teng Yu, Yuya Ebihara, and the broader Trino community to finally contribute the long-coveted Trino Snowflake connector. It is a success story paired with a blueprint for individuals and corporations wanting to contribute to open source projects they use. These stories are valuable in that they demonstrate how to be most effective in collaborating with strangers-soon-to-be-friends and common pitfalls to avoid.
A common challenge in open source
Despite the importance of delivering marketing and education in a community (aka edutainment), it’s only the first part of the equation of what makes open source projects successful. Once developers see some exciting video or tutorial, they ultimately land on the docs site, GitHub, StackOverflow, or some communication platform in the community. It's at this point that developers can easily lose the motivation if the docs lacks proper getting started materials or the community is completely silent. This is how I categorize the developer experience (aka devex), which aims to improve both the user and contributor experiences in the developer community by empowering decisions through hands-on learning, removing inefficiencies, and as we'll cover here, exposing untapped opportunities.
Much like any open source project, maintainers on the Trino project struggle with communicating the lack of proper resources to build and test new features built for various proprietary software. For those less familiar, Trino is a federated query engine with multiple data sources. Trino tests integrations with open data sources by running small local instances of the connecting system. Snowflake is a proprietary, cloud-native data warehouse, also known as cloud data platform. This provided no viable and free way to support testing this integration that was eagerly sought by many. After an initial attempt by my friend Phillipe Gagnon, a similar pattern emerged with the second pull request where the development velocity started strong and after some months stagnated.
Cognitive surplus and communication deficit
A common and unfortunate class of issues are that various well-known larger objectives known among the core group often move faster than less-established individual contributions. These additions are often much needed and welcomed, but often fail to fit a larger project roadmap narrative. As its easier to coordinate between the smaller core group as trust and norms have been communicated and established. This makes changes outside of this group have a higher likelihood to get lost in the shuffle. As an open source project grows, you end up with a cognitive surplus in the form of an abundance of bright people willing to share their time, intellect, and experience with a larger community.