Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 34 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,58 +2,59 @@

This page will walk you through using OpenTofu to manage your Marqo Cloud resources.

OpenTofu, a fork of Terraform, is an open-source infrastructure as code tool that enables you to safely and predictably create, change, and delete cloud resources.
For more information on opentofu and terraform, please visit the following links: [OpenTofu](https://opentofu.org) and [Terraform](https://www.terraform.io/).
OpenTofu, a fork of Terraform, is an open-source infrastructure as code tool that enables you to safely and predictably create, change, and delete cloud resources.
For more information on OpenTofu and Terraform, please visit the websites for [OpenTofu](https://opentofu.org) and [Terraform](https://www.terraform.io/).

The marqo opentofu provider is located at [`registry.opentofu.org/marqo-ai/marqo`](https://github.com/opentofu/registry/blob/main/providers/m/marqo-ai/marqo.json).
The Marqo OpenTofu provider is located at [`registry.opentofu.org/marqo-ai/marqo`](https://github.com/opentofu/registry/blob/main/providers/m/marqo-ai/marqo.json).

If you wish to use the terraform provider instead, please note the following
- replace the provider source with `marqo-ai/marqo`
- replace all `tofu` commands in the guide below with `terraform`
If you wish to use the Terraform provider instead, please note the following:
- replace the `provider` source with `marqo-ai/marqo`
- replace all `tofu` commands below with `terraform`

Detailed documentation can be found in marqodocs [here](https://docs.marqo.ai/2.11/Cloud-Reference/opentofu_provider/).
Detailed documentation can be found in [the Marqo docs](https://docs.marqo.ai/latest/reference/cloud/opentofu-provider/).

---

## Installation Instructions

1. Install Opentofu by following the instructions on the [Opentofu website](https://opentofu.org/docs/intro/install/). Alternatively, install Terraform by following the instructions on the [Terraform website](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli).
2. Create a Marqo configuration (a `.tf` file with your marqo API and endpoint details) in a new directory
1. Install OpenTofu by following the instructions on the [OpenTofu website](https://opentofu.org/docs/intro/install/) (or Terraform via the [Terraform website](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)).
2. Create a `.tf` configuration file with your Marqo API and endpoint details (see the examples below).
3. Run `tofu init` in the directory you created the configuration in to initialize the configuration
4. At this point, you should be able to run `tofu plan` and `tofu apply` to create and manage your Marqo resources.

Some common commands are:
- `tofu plan` - checks whether the configuration is valid and what actions will be taken
- `tofu apply` - creates or updates the resources in your account
- `tofu destroy` - deletes the resources in your account
- `tofu plan`: checks whether the configuration is valid and what actions will be taken
- `tofu apply`: creates or updates the resources in your account
- `tofu destroy`: deletes the resources in your account

See the [Opentofu documentation](https://opentofu.org/docs/intro/) for more information on how to use Opentofu.
See the [OpenTofu documentation](https://opentofu.org/docs/intro/) for more information on how to use OpenTofu.

## Overview of Features

The marqo opentofu provider supports the following:
The Marqo OpenTofu provider supports the following features:

- A datasource called `marqo_read_indices` that allows you to read all of your marqo indexes in your account.
- A resource called `marqo_index` that allows you to create and manage a marqo index.
- A datasource called `marqo_read_indices` that allows you to describe all of the indexes in your Marqo Cloud account.
- A resource called `marqo_index` that allows you to create and manage a Marqo Cloud index.

## Sample Configuration

For both of the examples below, create a file within each configuration directory named `terraform.tfvars` containing your api key as follows

```python
```
marqo_api_key = "<KEY>"
```

Note that the host must be set to `"https://api.marqo.ai/api/v2"`

### Reading All Indexes in Your Account (datasource)

Note that the host must be set to `"https://api.marqo.ai/api/v2"`

```terraform
terraform {
required_providers {
marqo = {
source = "registry.opentofu.org/marqo-ai/marqo"
version = "1.0.1"
version = "1.2.0"
}
}
}
Expand All @@ -79,12 +80,14 @@ variable "marqo_api_key" {

### Creating and Managing a Structured Index (resource)

Refer to the [provider documentation](https://docs.marqo.ai/latest/reference/cloud/opentofu-provider) for more information on the settings available for the `marqo_index` resource.

```terraform
terraform {
required_providers {
marqo = {
source = "registry.opentofu.org/marqo-ai/marqo"
version = "1.0.1"
version = "1.2.0"
}
}
}
Expand All @@ -95,7 +98,7 @@ provider "marqo" {
}

resource "marqo_index" "example" {
index_name = "example_index_dependent_2"
index_name = "example_index"
settings = {
type = "structured"
vector_numeric_type = "float"
Expand All @@ -112,13 +115,13 @@ resource "marqo_index" "example" {
},
],
number_of_inferences = 1
inference_type = "marqo.CPU.large"
storage_class = "marqo.basic"
number_of_shards = 1
number_of_replicas = 0
number_of_shards = 2
tensor_fields = ["multimodal_field"],
model = "open_clip/ViT-L-14/laion2b_s32b_b82k"
tensor_fields = ["multimodal_field"],
normalize_embeddings = true
inference_type = "marqo.CPU.small"
text_preprocessing = {
split_length = 2
split_method = "sentence"
Expand All @@ -143,6 +146,12 @@ output "created_index" {

variable "marqo_api_key" {
type = string
description = "Marqo API key"
description = "Marqo Cloud API key"
}
```

## Testing

To verify that your index has been created and is ready to use, you can add some data to the index and query it using the Marqo API.

Browse to your index details page in the Marqo Cloud Console for snippets to help you get started.
4 changes: 2 additions & 2 deletions examples/create_index/create_index.tf
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ resource "marqo_index" "example" {
treat_urls_and_pointers_as_images = true
model = "open_clip/ViT-L-14/laion2b_s32b_b82k"
normalize_embeddings = true
inference_type = "marqo.CPU.small"
inference_type = "marqo.CPU.large"
all_fields = []
number_of_inferences = 1
number_of_replicas = 0
Expand Down Expand Up @@ -49,4 +49,4 @@ output "created_index" {
variable "marqo_api_key" {
type = string
description = "Marqo API key"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ resource "marqo_index" "example" {
trust_remote_code = false
}
normalize_embeddings = false
inference_type = "marqo.CPU.small"
inference_type = "marqo.CPU.large"
all_fields = []
number_of_inferences = 1
number_of_replicas = 0
Expand Down Expand Up @@ -56,4 +56,4 @@ output "created_index" {
variable "marqo_api_key" {
type = string
description = "Marqo API key"
}
}
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
terraform {
required_providers {
marqo = {
source = "marqo/marqo"
#version = "1.0.1"
source = "marqo/marqo"
version = "1.2.0"
}
}
}
Expand Down Expand Up @@ -74,4 +74,4 @@ output "created_index" {
variable "marqo_api_key" {
type = string
description = "Marqo API key"
}
}
6 changes: 3 additions & 3 deletions examples/create_structured_index/create_structured_index.tf
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ terraform {
required_providers {
marqo = {
source = "marqo/marqo"
version = "1.0.1"
version = "1.2.0"
}
}
}
Expand Down Expand Up @@ -36,7 +36,7 @@ resource "marqo_index" "example" {
tensor_fields = ["multimodal_field"],
model = "open_clip/ViT-L-14/laion2b_s32b_b82k"
normalize_embeddings = true
inference_type = "marqo.CPU.small"
inference_type = "marqo.CPU.large"
text_preprocessing = {
split_length = 2
split_method = "sentence"
Expand All @@ -62,4 +62,4 @@ output "created_index" {
variable "marqo_api_key" {
type = string
description = "Marqo API key"
}
}
2 changes: 1 addition & 1 deletion examples/read_datasource/read_datasource_indices.tf
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ output "indices_in_marqo_cloud" {
variable "marqo_api_key" {
type = string
description = "Marqo API key"
}
}
6 changes: 3 additions & 3 deletions internal/provider/indices_datasource_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ func testAccDataSourceIndexConfig(name string) string {
treat_urls_and_pointers_as_media = true
model = "open_clip/ViT-L-14/laion2b_s32b_b82k"
normalize_embeddings = true
inference_type = "marqo.CPU.small"
inference_type = "marqo.CPU.large"
all_fields = []
number_of_inferences = 1
number_of_replicas = 0
Expand All @@ -95,7 +95,7 @@ func testAccDataSourceIndexConfig(name string) string {
}
filter_string_max_length = 20
}
}
}
`, name)
}

Expand All @@ -115,7 +115,7 @@ func testAccCheckIndexInDataSource(dataSourceName string, indexName string) reso
if ds.Primary.Attributes[fmt.Sprintf("items.%d.index_name", i)] == indexName {
// Check attributes for this specific index
attributesToCheck := map[string]string{
"inference_type": "marqo.CPU.small",
"inference_type": "marqo.CPU.large",
"storage_class": "marqo.basic",
// Add more attributes to check
}
Expand Down
6 changes: 3 additions & 3 deletions internal/provider/indices_resource_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ func TestAccResourceStructuredIndex(t *testing.T) {
resource.TestCheckResourceAttr("marqo_index.test", "settings.vector_numeric_type", "float"),
resource.TestCheckResourceAttr("marqo_index.test", "settings.model", "open_clip/ViT-L-14/laion2b_s32b_b82k"),
resource.TestCheckResourceAttr("marqo_index.test", "settings.normalize_embeddings", "true"),
resource.TestCheckResourceAttr("marqo_index.test", "settings.inference_type", "marqo.CPU.small"),
resource.TestCheckResourceAttr("marqo_index.test", "settings.inference_type", "marqo.CPU.large"),
resource.TestCheckResourceAttr("marqo_index.test", "settings.number_of_inferences", "1"),
resource.TestCheckResourceAttr("marqo_index.test", "settings.number_of_replicas", "0"),
resource.TestCheckResourceAttr("marqo_index.test", "settings.number_of_shards", "2"),
Expand Down Expand Up @@ -202,7 +202,7 @@ func testAccResourceStructuredIndexConfig(name string) string {
tensor_fields = ["multimodal_field"]
model = "open_clip/ViT-L-14/laion2b_s32b_b82k"
normalize_embeddings = true
inference_type = "marqo.CPU.small"
inference_type = "marqo.CPU.large"
text_preprocessing = {
split_length = 2
split_method = "sentence"
Expand Down Expand Up @@ -325,7 +325,7 @@ func testAccResourceIndexConfigCustomModel(name string) string {
type = "hf"
}
normalize_embeddings = true
inference_type = "marqo.CPU.small"
inference_type = "marqo.CPU.large"
number_of_inferences = 1
number_of_replicas = 0
number_of_shards = 1
Expand Down
Loading