Posted on October 2, 2018 by Brian Jones
A comment I wrote on r/haskell in reply to “How to introduce Haskell to a webdev shop?” prompted this post, which I hope ends up being helpful to others who may be considering introducing Haskell to their team.
This post makes the assumption that the reader understands why they themselves would want to use Haskell and what its benefits are, so instead focuses on how we’ve made it a successful part of our own company’s development culture.
First of all I think it’s important to understand the context. Where we came from, what we do. Hopefully this gives the reader reasonable expectations about what is written here.
AlasConnect is a data center company which was founded in the 1990s in Fairbanks, Alaska under the wing of Golden Valley Electric Association. Since then it has expanded out to also become a Microsoft focused Managed Service Provider that does server hosting and tech support for small-medium companies across the Fairbanks, Anchorage, and Palmer areas. A few years ago it was bought by Matanuska Telephone Association, an internet company which services part of Southern Alaska, and AlasConnect now handles all of their internal IT support as well.
Since 1999 I’ve been loosely associated with AlasConnect in various capacities and have maintained a great relationship with the CEO. Two years ago I was asked to come back and work for them to lead their new software development group, a combination of developers from both AlasConnect and MTA. Prior to that AlasConnect did not really have a development group to speak of, during which time my colleague Chris and I had worked for them as software consultants.
This is all important to note because it shows that a relationship built on understanding of the company, its people, and its goals was established long before the development group was created. Without this, strolling in without knowing anything about the company and its existing environment and introducing Haskell would have more than likely been impossible.
I believe another reason it’s worth pointing all of this out is because we aren’t a singularly focused startup working on a single product. We are a small embedded development team working on a handful of applications spread across multiple companies, which means the amount of jumping around, maintenance, and outright non-standard applications we write are across the board on any given month. This type of scenario plays very well into Haskell’s maintenance and refactoring story.
After having used several common languages in production settings and seeing the myriad of ways they could fail, I became obsessed with finding a language that offered strong type safety and allowed one to write concise and easy to maintain production code. After watching a 2013 QuakeCon talk by John Carmack in which he mentioned Haskell, and how if he could go back and redo it all he’d pick an FP language, I knew what I had to do.
In 2014 I started using Haskell for various personal projects, and even introduced it as a minor side project at a Japanese game company I worked for. AlasConnect ended up being the perfect opportunity to make an attempt at introducing it as an official development language. It’s not often that one gets the chance to build a brand new development culture from scratch! Nor is it a decision that should be made on a whim. If you cannot come up with solid business reasons beyond, “it’s a cool programming language and I want to play with it in a production setting” then I would suggest you reconsider what your goals are.
These are the points from my r/haskell post, however, I should point out that they are mostly being made in a team setting. Solo/siloed developers will obviously have a completely different take on some of these points.
- You need to have enough clout to make it happen. You are a lead, or a manager, and have your immediate manager’s ear and their blessing. This will be difficult to do otherwise.
- You need to expect pushback.
- Haskell is so incredibly different than (insert imperative language here) people will not be productive for months.
- This timeline could go longer if you yourself do not understand Haskell well enough.
- If your company can afford the time invested in the learning phase, you may be successful in this endeavor.
- You have to be the one to take initiative and build the first application, and then train people into it. A bunch of newbies to the language left floundering will start to resent it.
- Book resources are great, but they aren’t enough. You absolutely need a project for people to hack on so they feel productive.
- You have to be diplomatic and understanding. Frustrated coworkers are the fastest way to make them hate Haskell for life.
- Remember, it’s not just the language, but the ecosystem. What is the difference between cabal, stack, and nix? Hackage and stackage? Tools like hoogle exist.
- Can you explain Haskell’s warts? Nothing is perfect, and being able to show everyone that Haskell has bad points actually goes a long way to helping ground them.
Immediately after signing all of my paperwork and officially becoming an employee I started with getting the blessing of our CTO. After explaining how a strongly type language like Haskell could erase a large class of bugs my colleague Chris and I have experienced over the many years we’ve been at this, but could also conceivably take more time to train people up on as the ultimate price for using a language so different from standard imperative programming, he had no issues with it and was on board.
The AlasConnect + MTA Team
When I started nearly two years ago AlasConnect itself had no standard development tooling or culture, and were just getting started with spinning up their own projects. The small group of members sounded interested in functional programming and were exploring Elixir at the time.
The MTA group was using C#.NET, which makes sense given the enterprise nature of the company. They had a small handful of applications they were writing and supporting at the time.
Needless to say, there was a strong “us vs. them” vibe at the start. In order to get everyone working together I decided to do the following:
- Use all inclusive team pronouns. Rather than say, “the MTA group needs to do X”, instead, “we need to focus on MTA task X”
- Put all of our new projects in a monorepo so that everyone is loosely staring at everyone else’s code
- This point became more important as everyone started identifying common libraries to break out and share between projects
- Assign at least 2 people to any project, further erasing any AC/MTA boundaries, as well as get rid of siloed-developer mentality
That’s the big question of course, and it spawns a lot of sub-questions.
- What are the benefits of using Haskell period?
- Why would AlasConnect itself choose Haskell?
- Why would MTA choose Haskell if it’s already established with using C#?
Benefits of Using Haskell
To kick things off I wrote some slides to help explain the power of the language, although they feel slightly out of date now. The main point was to address some concerns everyone may have, and above all discuss some of the benefits of using Haskell.
I think the initial points are still valid.
- Reduce programmer cognitive overload
- Produce easy to reason about code
- Focus on the small, not the large
- Reduce the number of bugs with strong compile time checks
- Prefer catching problems at compile time
- Reduce testing surface
- Utilize strong types to build correct software
- Describe complex domains
- Runtime guarantees
- Ease of long term maintenance
- Refactor heavily without fear
- Types help guide compilation
There were some rumors. “I heard it…”
- Is a math language
- I have to learn lambda calculus (and category theory) to even use it
- Is a research language
- Only language researchers use it
- Is not used in production systems
- Is incredibly hard to learn
- Uses crazy terms like Monoid, Functor, Applicative, Monad
And practical concerns.
- Not a corporate backed programming language (like C#, Java, Go, Erlang, etc)
- High learning curve
- If I don’t grasp it in a day I will never be productive
- Smaller community compared to other popular languages
Time was spent talking about both the benefits and the drawbacks. As I mentioned earlier, being able to explain the warts of your chosen language helps ground people. If all you talk about are the ivory tower benefits, and then one day your team gets blindsided by things like terrible record syntax or poor documentation, they will likely begin to resent your choice.
Haskell at AlasConnect
Our primary mission goes beyond writing non-marketable software for the companies we support. One of the long term goals the CEO and I have slowly worked toward over the years is to put ourselves in a position to break out and write our own custom software. The first step of course, was to lock in our development culture, processes, and tooling.
As I came on board the AC team was talking about using Elixir, a tool Chris had been utilizing for the past year and was somewhat happy with. (I recall him saying, “There is no way you will ever get me to write Haskell professionally” prior to me signing on. He is now our biggest Reflex advocate). Elixir/Erlang is definitely not a bad choice, but I spent some time showing people the merits of a strongly typed language like Haskell, and the benefits of having the compiler catch bugs long before the code ever reached production. The most salient point was code sharing: The ability to use a cross section of code between a backend API and frontend SPA, which removes a large class of problems we had experienced when writing applications in two disparate languages in the past (Go and Angular.js).
Haskell at MTA
This was slightly harder. There was a clear split between the AC and MTA teams. Because the MTA developers were happy with writing things in C# it made little sense to completely sway them away from it, so we decided to let them focus on MTA-specific apps in C#. It was still important to train them on the tooling AC standardized on though, because as AC employees they could very well be working on non-MTA projects in the future. This was met with varying levels of resistance.
Eventually a few people unhappy with the merger ended up leaving, and I was told to assign everyone to the core MTA project. The AC developers had no interest in learning C#.NET, even though the remaining MTA developers already had a small start on the project in C#. This was ultimately the tipping point which got everyone using Haskell, a simple case of attrition and resource reallocation. We restarted the project in Haskell and haven’t looked back since.
Does it make sense to use Haskell in an enterprise level company like MTA? I would argue yes.
The majority of our work involves:
- Rewriting a public facing customer self service portal (billing, internet upgrades, etc)
- Writing new in-house tools which provide better UI/UX to processes which have been very manual up to this point
- Writing a large number of microservices which wrap some blackbox vendor APIs that are incredibly hard to work with directly
- Potentially writing some software which interacts directly with network devices over telnet, a place Haskell can truly shine (parsers!)
Honestly, either tool would be fine in most of these cases. However, the fact that the entire team is on the same page regardless of which company’s software they are working on is the biggest payoff of all. Now we are not limited by resource constraints based on primary tooling knowledge between two loosely coupled teams, we are simply “The Team”.
It wasn’t just Haskell
I should point out that our primary Operations guru and I have pretty much seen eye to eye on things this whole time, and that having a solid set of modern day DevOps practices also helped get everyone on the same page.
- Everyone started seeing the benefits of GitHub PRs and code reviews, something they had never done before
- Everyone clearly saw the benefits of a CI server catching build errors long before the code reached production
- This is a great case for picking a compiled language, half your problems can be caught with a compiler instead of tests
- Good service orchestration allowed us to make a solid microservice architecture instead of picking something like a heavyweight Enterprise Service Bus solution
- Once the MTA developers saw the flexibility here, it felt like it became much easier to bring them over to Haskell
- A monorepo, while not entirely necessary, has all of us looking at the Big Picture as we build and release new applications and common libraries
u/theindigamer asked a great set of followup questions in my r/haskell post, which I will attempt to give better answers to here.
- How long did it take to go from zero Haskell to reasonably comfortable (mean +- stdev)?
- How did you manage different people learning at a different pace?
- Were people incentivized/nudged to learn Haskell? If yes, how?
- More generally, how did you offset the negative feelings people felt when they were unproductive as beginners? How did you manage expectations w.r.t. “yes, this will take more time to learn than you might anticipate”?
- How did you bootstrap, so to speak? By writing some non-critical code in Haskell? At the bootstrapping step, I’d think it would be hard to advertise ease of refactoring and have people take it seriously.
It probably helps to understand that we didn’t take all of our developers, drop everything, and then spend 6+ months learning Haskell while doing absolutely nothing else. That is a recipe for angry execs and unhappy customers, and a route we definitely did not go down. Instead, Chris and I started prototyping Servant + Reflex while everyone else continued working on their current projects, and once we reached that tipping point I mentioned earlier we immediately switched gears and started implementing a concrete project on top of our work up to that point. Everything has continued to evolve, of course, as it will, but by putting in that initial effort we’ve helped set the base structure for everything else we plan on doing in the near future.
How long did it take to go from zero Haskell to reasonably comfortable (mean +- stdev)?
I spent a couple months prior to officially signing on with AlasConnect tinkering with Servant and Reflex, because I knew they were going to be core tools we would use, and I wanted to be somewhat comfortable with using them before introducing a full team to the language and ecosystem. I also spent time compiling a list of useful resources such as books, relevant blog posts, and other web resources that could help the team learn.
The variation between going from zero to somewhat-productive in Haskell for everyone has been between 1-2 months, and seems to depend on prior development experience, temperament, interest, learning style, ability to unlearn imperative principles, and above all the ability to query mentors. When we first started almost 2 years ago Chris and I were a little shaky with our mentoring, these days the cumulative knowledge of the team is making it much easier to train developers new to the language.
How did you manage different people learning at a different pace?
Everyone learns differently. After throwing people at LYAH or HB, we then spend a solid day explaining our entire ecosystem (there’s a good chunk of non-Haskell specific stuff out there to understand), as well as some Haskell fundamentals and how it applies to our long term vision. From there, we have been assisting them with knocking out small features on existing projects.
That said, we did throw our resident intern off the deep end. He is implementing a brand new application from scratch (using our toolset) under the guidance of a mentor.
Were people incentivized/nudged to learn Haskell? If yes, how?
I think adding my own experience here may help. I feel that being as patient and understanding as possible, explaining things often, and doing pair programming went a long way towards getting people more comfortable with the language. Throwing someone Haskell Book and then expecting them to wade through ~1200 pages of college text book-like material on their own and coming out the other end as pros is unreasonable.
My biggest mistake was in the very beginning, where I asked people to attempt to write a small relevant tool in the language. They barely understood the concepts, if at all, let alone the ecosystem, the tooling (stack, cabal, nix), where the documentation was (if at all), or even knowing something like Hoogle existed. This was quickly adding up to be a recipe for disaster, but was remedied by Chris and I building the base for a core project and adding people after the fact instead.
More generally, how did you offset the negative feelings people felt when they were unproductive as beginners? How did you manage expectations w.r.t. “yes, this will take more time to learn than you might anticipate”?
This was probably the hardest hurdle to get over. For the few that knew absolutely nothing about Haskell, it was frustrating to learn that they couldn’t pick up the language in a day, let alone a week. There were a few strong rants against Haskell and its community at some point, however, I think the desire to not be “defeated” allowed them to overcome their misgivings.
I think the biggest thing that has helped is, again, giving people simple features to hack on with an existing project. This gives them the opportunity to examine the code written by others, and at the very least monkey-see monkey-do until they internalize some of Haskell’s core concepts. This is still a tough pill for some to swallow, some people don’t want to simply replicate how existing code works at first, they want to understand the underlying mechanics. This leads them down the rabbit hole, and ultimately to despair (imagine leaving someone to implement a basic lens, and when you turn back around they are reading books on category theory to understand the underlying code Edward Kmett wrote). I try to carefully explain that once the concepts become internalized things get easier, and gently guide them back to their primary goal.
In the beginning when we had absolutely nothing this was a little harder to deal with, but now that we have a large number of solid projects the concern is much lower these days.
The recognition of the power of refactoring began to occur after we put that initial base code in place, and still continues to amaze people today. Recently someone said to me after I dropped a monster multi-project commit in place due to a change in a shared library, “If I had been doing this in Python I would have just scrapped it and wrote a v2, because there is no way I could have sanely refactored that many modules without breaking the entire thing.”
There’s no one-size-fits-all solution here. Successfully introducing Haskell into a company can be both challenging and rewarding.
While I did suggest that you need to be a lead or a manager to make it happen, people with less leverage can still hook the language in via smaller side projects and build up consensus that way. Everyone’s situation is different, but if you attempt to introduce it to a team then it should be made clear that soft skills are one of the most important factors. Your ability to help guide and mentor your team will go a long way towards your eventual success.
On occassion I post on twitter, mostly about Haskell/technical things. Feel free to follow if you want to hear more about a company using Haskell in Alaska.