Terraform cannot init custom provider written in Go

I am trying to do this tutorial to create a custom terraform module using plugin framework.

Everything is fine up to the point where I need to use terraform to install & verify the provider. It fails with:

➜  examples git:(main) ✗ terraform init
Initializing the backend...
Initializing provider plugins...
- Finding latest version of example.com/vladmiller/example...
│ Warning: Provider development overrides are in effect
│
│ The following provider development overrides are set in the CLI configuration:
│  - example.com/vladmiller/example in /Users/vladi/go/bin
│
│ Skip terraform init when using provider development overrides. It is not necessary and may
│ error unexpectedly.
╵
╷
│ Error: Invalid provider registry host
│
│ The host "example.com" given in in provider source address "example.com/vladmiller/example"
│ does not offer a Terraform provider registry.

I also tried:

• cloning repo from tutorial, but it also fails
• using locally installed Go instead of GoLand
• go1.18.8 instead of go1.19.x
• pointing folder in .terraformrc to a folder where I develop code (I thought that maybe terraform-registry-manifest.json needs to be in the same folder w/ binary)
• cloning and trying this repo – no luck
• I'm currently trying to comprehend terraform code around how they load plugins...

Wondering if anyone have any ideas how to troubleshoot. Thank you in advance.

Looking at the logs it's clear that terraform is trying to get registry details by making request to example.com. Which, I expect, it shouldn't do because there are dev_overrides.

...
Service discovery for example.com at https://example.com/.well-known/terraform.json
...

Below is are more logs and example of my setup.

➜  examples git:(main) ✗ TF_LOG=trace terraform init
2022-11-10T16:39:54.255+0200 [INFO]  Terraform version: 1.3.4
2022-11-10T16:39:54.255+0200 [DEBUG] using github.com/hashicorp/go-tfe v1.9.0
2022-11-10T16:39:54.255+0200 [DEBUG] using github.com/hashicorp/hcl/v2 v2.14.1
2022-11-10T16:39:54.255+0200 [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2022-11-10T16:39:54.255+0200 [DEBUG] using github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734
2022-11-10T16:39:54.255+0200 [DEBUG] using github.com/zclconf/go-cty v1.12.0
2022-11-10T16:39:54.255+0200 [INFO]  Go runtime version: go1.19.3
2022-11-10T16:39:54.255+0200 [INFO]  CLI args: []string{"terraform", "init"}
2022-11-10T16:39:54.255+0200 [TRACE] Stdout is a terminal of width 206
2022-11-10T16:39:54.255+0200 [TRACE] Stderr is a terminal of width 206
2022-11-10T16:39:54.255+0200 [TRACE] Stdin is a terminal
2022-11-10T16:39:54.255+0200 [DEBUG] Attempting to open CLI config file: /Users/vladi/.terraformrc
2022-11-10T16:39:54.255+0200 [INFO]  Loading CLI configuration from /Users/vladi/.terraformrc
2022-11-10T16:39:54.255+0200 [DEBUG] Explicit provider installation configuration is set
2022-11-10T16:39:54.255+0200 [TRACE] Selected provider installation method cliconfig.ProviderInstallationDirect with includes [] and excludes []
2022-11-10T16:39:54.255+0200 [INFO]  CLI command args: []string{"init"}

Initializing the backend...
2022-11-10T16:39:54.256+0200 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2022-11-10T16:39:54.256+0200 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2022-11-10T16:39:54.256+0200 [DEBUG] New state was assigned lineage "b3cc2d29-4dd2-a955-32f4-55d7481d5445"
2022-11-10T16:39:54.256+0200 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2022-11-10T16:39:54.256+0200 [TRACE] Meta.Backend: instantiated backend of type <nil>
2022-11-10T16:39:54.256+0200 [DEBUG] Provider example.com/vladmiller/example is overridden by dev_overrides
2022-11-10T16:39:54.256+0200 [DEBUG] Provider example.com/vladmiller/example is overridden to load from /Users/vladi/go/bin
2022-11-10T16:39:54.256+0200 [DEBUG] checking for provisioner in "."
2022-11-10T16:39:54.263+0200 [DEBUG] checking for provisioner in "/opt/homebrew/bin"
2022-11-10T16:39:54.263+0200 [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2022-11-10T16:39:54.263+0200 [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2022-11-10T16:39:54.263+0200 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2022-11-10T16:39:54.263+0200 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2022-11-10T16:39:54.263+0200 [TRACE] statemgr.Filesystem: read nil snapshot
2022-11-10T16:39:54.263+0200 [DEBUG] Provider example.com/vladmiller/example is overridden by dev_overrides

Initializing provider plugins...
- Finding latest version of example.com/vladmiller/example...
2022-11-10T16:39:54.263+0200 [DEBUG] Service discovery for example.com at https://example.com/.well-known/terraform.json
2022-11-10T16:39:54.263+0200 [TRACE] HTTP client GET request to https://example.com/.well-known/terraform.json
╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI configuration:
│  - example.com/vladmiller/example in /Users/vladi/go/bin
│ 
│ Skip terraform init when using provider development overrides. It is not necessary and may error unexpectedly.
╵

╷
│ Error: Invalid provider registry host
│ 
│ The host "example.com" given in in provider source address "example.com/vladmiller/example" does not offer a Terraform provider registry.
╵

➜  examples git:(main) ✗ terraform version
Terraform v1.3.4
on darwin_arm64

➜  examples git:(main) ✗ cat ~/.terraformrc
provider_installation {

  dev_overrides {
      "example.com/vladmiller/example" = "/Users/vladi/go/bin/"
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

➜  examples git:(main) ✗ tree /Users/vladi/go/bin
/Users/vladi/go/bin
└── terraform-provider-example

0 directories, 1 file

➜  bin ./terraform-provider-example 
This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically

➜  terraform-provider-example git:(main) ✗ cat main.go
package main

import (
	"context"
	"flag"
	"log"

	"github.com/hashicorp/terraform-plugin-framework/providerserver"
	"terraform-provider-example/example"
)

var (
	// these will be set by the goreleaser configuration
	// to appropriate values for the compiled binary
	version string = "dev"

	// goreleaser can also pass the specific commit if you want
	// commit  string = ""
)

func main() {
	var debug bool

	flag.BoolVar(&debug, "debug", false, "set to true to run the provider with support for debuggers like delve")
	flag.Parse()

	err := providerserver.Serve(context.Background(), example.New(version), providerserver.ServeOpts{
		Address: "example.com/vladmiller/example",
		Debug:   debug,
	})
	if err != nil {
		log.Fatal(err.Error())
	}
}

➜  examples git:(main) ✗ cat provider-verification.tf 
terraform {
  required_providers {
    example = {
      source = "example.com/vladmiller/example"
    }
  }
}

provider "example" {}

➜  terraform-provider-example git:(main) ✗ cat go.mod 
module terraform-provider-example

go 1.18

require github.com/hashicorp/terraform-plugin-framework v0.15.0

require (
        github.com/fatih/color v1.13.0 // indirect
        github.com/golang/protobuf v1.5.2 // indirect
        // truncated
)

UPDATE 1: I got wondering that maybe there is an issue with my environment. I'm on MacOS Ventura 13.0 w/ M2.

I started a container from ubuntu/latest aarch64 and installed latest terraform in the container. Cloned provider branch of https://github.com/hashicorp/terraform-provider-hashicups-pf.
And it's still the same issue.

I have also tried terraform 1.3.0, 1.2.0 and 0.15.5 just in case.

root@23de1551280a:~/terraform-provider-hashicups-pf/examples/provider-install-verification# TF_LOG=debug terraform init
2022-11-11T12:45:23.284Z [INFO]  Terraform version: 1.3.4
2022-11-11T12:45:23.284Z [DEBUG] using github.com/hashicorp/go-tfe v1.9.0
2022-11-11T12:45:23.284Z [DEBUG] using github.com/hashicorp/hcl/v2 v2.14.1
2022-11-11T12:45:23.284Z [DEBUG] using github.com/hashicorp/terraform-config-inspect v0.0.0-20210209133302-4fd17a0faac2
2022-11-11T12:45:23.284Z [DEBUG] using github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734
2022-11-11T12:45:23.284Z [DEBUG] using github.com/zclconf/go-cty v1.12.0
2022-11-11T12:45:23.284Z [INFO]  Go runtime version: go1.19.3
2022-11-11T12:45:23.284Z [INFO]  CLI args: []string{"terraform", "init"}
2022-11-11T12:45:23.284Z [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2022-11-11T12:45:23.284Z [INFO]  Loading CLI configuration from /root/.terraformrc
2022-11-11T12:45:23.285Z [DEBUG] Explicit provider installation configuration is set
2022-11-11T12:45:23.286Z [INFO]  CLI command args: []string{"init"}

Initializing the backend...
2022-11-11T12:45:23.287Z [DEBUG] New state was assigned lineage "91db86b6-1777-8a07-27fd-aac42c238c35"
2022-11-11T12:45:23.287Z [DEBUG] Provider hashicorp.com/edu/hashicups-pf is overridden by dev_overrides
2022-11-11T12:45:23.287Z [DEBUG] Provider hashicorp.com/edu/hashicups-pf is overridden to load from /root/go/bin
2022-11-11T12:45:23.287Z [DEBUG] checking for provisioner in "."
2022-11-11T12:45:23.287Z [DEBUG] checking for provisioner in "/root"
2022-11-11T12:45:23.287Z [DEBUG] Provider hashicorp.com/edu/hashicups-pf is overridden by dev_overrides

Initializing provider plugins...
- Finding latest version of hashicorp.com/edu/hashicups-pf...
2022-11-11T12:45:23.287Z [DEBUG] Service discovery for hashicorp.com at https://hashicorp.com/.well-known/terraform.json
2022-11-11T12:45:23.377Z [DEBUG] Service discovery redirected to https://www.hashicorp.com/.well-known/terraform.json
╷
│ Warning: Provider development overrides are in effect
│
│ The following provider development overrides are set in the CLI configuration:
│  - hashicorp.com/edu/hashicups-pf in /root/go/bin
│
│ Skip terraform init when using provider development overrides. It is not necessary and may error unexpectedly.
╵

╷
│ Error: Invalid provider registry host
│
│ The host "hashicorp.com" given in in provider source address "hashicorp.com/edu/hashicups-pf" does not offer a Terraform provider registry.
╵

Update 2: at this point I am inclined to believe that there is something wrong with the repo as if it's missing something.

Update 3: So, terraform init does not work, however, terraform plan works like expected. :thinking_face:

I guess I'm unblocked to continue with the development process.
Reading the tutorial again the do not seem to do terraform init there.

The dev_overrides argument you mentioned tells other Terraform commands like terraform apply to ignore the dependency lock file and provider cache directory populated by terraform init and to use your specified path instead. It doesn't affect the behavior of terraform init itself; the installer will always try to find a reasonable package to download and record in the dependency lock file, to avoid creating an incomplete lock file.

As the warning message mentions, you don't actually need to run terraform init if your goal is only to test your overridden provider. If you just run terraform apply directly then your configured override will prevent Terraform from looking into the dependency lock file to find that provider and it'll instead just use directly the directory you specified. Automatic installation isn't necessary in this case because the package is already available at a designated location on your computer.

> Skip terraform init when using provider development overrides. It is not necessary and may error unexpectedly.