MemConn: an in-memory network stack for Go

By akutz

MemConn (github.com/akutz/memconn) is an in-memory network stack for Go that provides:

  • Support for named connections
  • Support for net.Conn‘s deadline semantics (supported natively in Go 1.10+)
  • Better performance than a TCP or UNIX domain socket
  • An ideal solution for writing tests for HTTP and gRPC applications

Please keep reading to discover what MemConn is, how to use it, and why.

What

The Golang net package revolves largely around the following two functions:

 // Dial connects to the address on the named network. func Dial(network, address string) (net.Conn, error) // Listen announces on the local network address. func Listen(network, address string) (net.Listener, error) 

MemConn provides its own Dial and Listen functions, both with signatures identical to their Golang counterparts. In addition to supporting the network types available in Go, MemConn’s Dial and Listen functions both accept a network type of memu – an in-memory, unbuffered network:

  1. Invoking memconn.Listen("memu", "MyNamedNetwork") initializes a new object that implements the net.Listener interface.
  2. The listener is cached by the provided address, `”MyNamedNetwork”.
  3. The listener’s Accept function ranges over a channel, listening for incoming connections.
  4. Calling memconn.Dial("memu", "MyNamedNetwork") looks up the cached listener by the provided name, "MyNamedNetwork".
  5. Two sides of a network connection are created with net.Pipe.
  6. One side of the connection is returned as the result of the preceeding memconn.Dial function.
  7. The other side is placed on the channel over which the listener’s Accept function ranges.

How

Developers familiar with Go’s net.Listen and net.Dial functions should have no issues using MemConn. The following code illustrates how to announce a new, unbuffered network named "net-00", and begins listening for incoming connections:

 lis, err := memconn.Listen("memu", "net-00") if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } for { conn, err := lis.Accept() if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } // Do something with the conn } 

Now that there is a server running any client can dial it:

 client, err := memconn.Dial("memu", "net-00") if err != nil { fmt.Fprintln(os.Stderr, err) os.Exit(1) } // Do something with the client 

For a full example with an HTTP server and client please see MemConn’s memconn_examples_test.go file.

Why

MemConn is the perfect candidate for writing Go tests that depend upon a net.Listener and need to scale. MemConn has no external dependencies and can scale as high as the memory available to the operating system. On the other hand:

  • TCP sockets can quickly exhaust the number of available files for the OS
  • UNIX domain sockets may pollute the user-visible filesystem and require cleanup

Additionally, MemConn is very performant compared to TCP and UNIX domain sockets:

Please see the files memconn_test.go and memconn_bench_test.go for more information on writing Go tests and benchmarks with MemConn.

Questions / Comments

Please sound off below or on GitHub with questions and comments!