golang API interface, what am I missing?

I want to create an interface to make it easy to add new storage backends.

package main

// Storage is an interface to describe storage backends
type Storage interface {
	New() (newStorage Storage)
}

// File is a type of storage that satisfies the interface Storage
type File struct {
}

// New returns a new File
func (File) New() (newFile Storage) {
	newFile = File{}
	return newFile
}

// S3 is a type of storage that satisfies the interface Storage
type S3 struct {
}

// New returns a new S3
func (S3) New() (newS3 S3) {
	newS3 = S3{}
	return newS3
}

func main() {
	// List of backends to choose from
	var myStorage map[string]Storage
	myStorage["file"] = File{}
	myStorage["s3"] = S3{}

	// Using one of the backends on demand
	myStorage["file"].New()
	myStorage["s3"].New()
}

But it seems not possible to define and satisfy a function that should return an object that satisfies the interface itself as well.

File.New() returns an object of type Storage which satisfies Storage.

S3.New() returns an object of type S3.
S3 should satisfies the interface Storage as well but I get this:

./main.go:32: cannot use S3 literal (type S3) as type Storage in assignment:
    S3 does not implement Storage (wrong type for New method)
        have New() S3
        want New() Storage

What am I doing wrong?
I hope I am missing something basic here.

This code does not make sense at all. You are either implementing a factory pattern which is tied to a struct which is of the type the factory is going to produce or you are reinventing the wheel in a wrong way by reimplementing the already existing new keyword and tie it to a struct which is nil the time you would use it.

You can either get rid of the helper function and simply use

s := new(S3)
f := new (File)

Or you could use a static Factory function like:

// Do NOT tie your Factory to your type
function New() S3 {
  return S3{}
}

Or, which seems to better suit your use case, create a factory interface, implement it and have its New() function return a Storage instance:

type StorageFactory interface {
  New() Storage
}

type S3Factory struct {}

function (f *S3Factory) New() Storage {
  return S3{}
}

There are various ways of registering your factory. You could use a global var and init

import "example.com/foo/storage/s3"

type FactoryGetter func() StorageFactory
type FactoryRegistry map[string] FactoryGetter

// Registry will be updated by an init function in the storage provider packages
var Registry FactoryRegistry

func init(){
  Registry = make(map[string] FactoryGetter)
}

// For the sake of shortness, a const. Make it abflag, for example
const storageProvider = "s3"

func main(){
  f := Registry[storageProvider]()
  s := f.New()
  s.List()
}

And somewhere in the S3 package

func init() {
  Registry["s3"] = function(){ return S3Factory{}}
}

You could even think of making the Factories taking params.

I like what you're doing here and I've actually worked on projects that involved very similar design challenges, so I hope my suggestions can help you out some.

In order to satisfy the interface, you'd need to update your code from...

// New returns a new S3
func (S3) New() (newS3 S3) {
    newS3 = S3{}
    return newS3
}

to this

// New returns a new S3
func (S3) New() (newS3 Storage) {
    newS3 = S3{}
    return newS3
}

This means you will receive an instance of Storage back, so to speak. If you want to then access anything from S3 without having to use type assertion, it would be best to expose that S3 function/method in the interface.

So let's say you want a way to List your objects in your S3 client. A good approach to supporting this would be to update Storage interface to include List, and update S3 so it has its own implementation of List:

// Storage is an interface to describe storage backends
type Storage interface {
    New() (newStorage Storage)
    List() ([]entry, error) // or however you would prefer to trigger List
}

...

// New returns a new S3
func (S3) List() ([] entry, error) {
    // initialize "entry" slice
    // do work, looping through pages or something
    // return entry slice and error if one exists
}

When it comes time to add support for Google Cloud Storage, Rackspace Cloud Files, Backblaze B2, or any other object storage provider, each of them will also need to implement List() ([] entry, error) as well - which is good! Once you've used this List function in the way you need, adding more clients/providers will be more like developing plugins than actually writing/architecting code (since your design is complete by that point).

The real key with satisfying interfaces is to have the signature match exactly and think of interfaces as a list of common functions/methods that you'd want every storage provider type to handle in order to meet your goals.

If you have any questions or if anything I've written is unclear, please comment and I'll be happy to clarify or adjust my post